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The purpose of this thesis is to study the usability an upgraded Maintenance Error 
Information Management System (MEIMS) tool used to capture human error in Naval 
Aviation maintenance mishaps. Built upon the Human Factors Analysis and 
Classification System-Maintenance Extension taxonomy, the tool provides the 
framework for examining maintenance errors that lead to mishaps, incidents, and 
personal injuries. The tool is developed for safety personnel, mishap investigators, 

Aircraft Mishap Board members, and analysts. In limited usability testing, the tool was 
found to be useful, but in need of revisions, specifically regarding functionality and user 
friendliness. Additionally, a tutorial is provided to better prepare targeted users of the 
tool. The study requires a review of mishap information systems, human error theories 
related to aviation mishaps, design considerations for human-computer interfaces and 
usability study applications. A follow-on usability study, conducted using two groups of 
potential users, one which received the tutorial and one which did not. It includes a 
survey regarding subjective responses about the prototype tool. The results indicate that 
the tutorial is effective in preparing and assisting potential users, and that the tool could 
make a significant impact in the reduction of mishap rates due to maintenance error. 
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I. INTRODUCTION 



A. OVERVIEW 

1. Background 

In the last half century, Naval Aviation has reduced the Class A Flight Mishaps 
(FM) rate significantly. In fact, the Class A FM rate was reduced by 50 percent in each 
decade from 1950 to 2000 (see Figure 1). Despite this success, the cost of mishaps, in 
terms of lives and assets, remains too high. Furthermore, human error, attributable to 
four out of every five mishaps, has remained constant (Nutwell & Sherman, 1997). In 
response to a 1996 F-14 Tomcat FM, which was largely attributable to human error, a 
Human Factors Quality Management Board (HFQMB) was initiated to study human error 
in Naval Aviation mishaps. The HFQMB’s charter (1997) was to reduce the mishap rate 
by identifying systematic improvements in processes and systems that affect human 
performance and to institutionalize continual improvements in these areas. The HFQMB 
is made up of Flag Officers, senior operational officers, and a cross section of 
Commanding Officers to seasoned naval aviators from all communities. Their objective 
was to reduce the Class A FM rate by half within three years, and by 75 percent within 10 
years (HFQMB, 1997). 

The HFQMB’s strategy entailed a three-pronged approach: (1) Mishap Data 
Analysis (MDA), (2) Organizational Benchmarking (OB), and (3) Command Safety 
Assessment (CSA). MDA studies historical records to determine established trends of 
human error in aviation mishaps. The Human Factors Analysis and Classification 
System (HFACS) was developed to accomplish this task. By examining and classifying 
past mishaps, prevailing human errors are prioritized and targeted for prevention. The 
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second phase, OB, involved the observance of other organizations, both military and 
civilian, for procedures and practices that might be useful in helping reduce the 
occurrence of human error. To this end, Naval Aviation adopted Organizational Risk 
Management, which associates conditions with risk and decision making methodology to 
reduce the occurrence of aviation mishaps, directly from the U. S. Army (Department of 
the Navy, 1997). CSA was developed at the Naval Postgraduate School by Civarelli and 
Figlock (1997) as a tool to measure the safety climate in a squadron from an aircrew 
perspective. It solicits opinions and attitudes towards procedures and practices to 
determine the effect of organizational and supervisory issues on flight safety (Nutwell & 
Sherman, 1997). 




Fiscal Year (FY) 



Figure 1: FY 1950-1999 Naval Aviation Class A Flight Mishap Rates 
(From School of Aviation Safety, 1999) 

In January 1999, the Vice Chief of Naval Operations extended the short-term goal 
of 50 percent reduction in Class A FM mishap rate from the end of FY 1999 to the end of 
FY2000. To meet this goal, the scope of the HFQMB needed to expand to include 
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human error in maintenance mishaps. Historically, approximately one in every five Class 
A FMs contains maintenance error. By expanding the focus to include maintenance 
error, it was believed that the 50 percent reduction objective could be met. An identical 
three-prong approach that had been used for aircrew error was utilized for maintenance 
error. 

2. Maintenance Mishap Data Analysis 

HFACS was developed as a framework for examining aircrew and supervisory 
error in Class A FMs. It concurs with the Naval Aviation Safety Program’s 
(OPNVAVINST 3750.6) notion of multiple causal factors, chain of events, and human 
factors. A maintenance extension (ME) was added to HFACS to meet the HFQMB’s 
expanded scope and was used to examine incidents and injuries (Schmidt, Figlock & 
Teeters, 1999), and major to minor mishap data (Schmidt, Schmorrow & Figlock, 2000). 
HFACS-ME was adopted for the upcoming revision of the Naval Aviation Safety 
Program. Furthermore, a laptop prototype tool, Maintenance Extension Information 
Management System (MEIMS) was developed to allow users to collect, catalog, collate 
and analyze mishap data, to better identify trends and for safety training. MEIMS, in 
limited usability testing, was determined to be a valuable tool that will provide the fleet 
great benefit (Wood, 2000). 

B. PROBLEM STATEMENT 

To meet the objective of a 50 percent reduction in Class A FMs, both aircrew and 
maintenance error needed to be targeted. The HFACS-ME taxonomy and MEIMS were 
developed to better analyze maintenance error. While proven useful in a limited usability 
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study, MEEMS is not without limitations. Users found the tool to be somewhat difficult 
to navigate, was lacking in assistance functionality and its interface needed to be more 
user-friendly. Additionally, it was determined that its intended users (e.g., squadron 
maintenance and safety personnel, squadron safety officers, Aircraft Mishap Board 
members, Naval Safety Center analysts, etc) would benefit from a training tutorial prior 
to accessing the tool (Wood, 2000). This study will further develop MEEMS based upon 
the initial usability study. It is believed that the fleet user, with proper training, should be 
able to easily access the tool and obtain useful information, which can then be used in 
training, hazard identification and trend analysis, to prevent future mishaps. 

This thesis investigates the following questions: 

1. What query capabilities are required by the users of the information 
system to best identify problem areas and trends? 

2. What additions will make the information system more user-friendly, 
allowing efficient and effective access to the system’s capabilities? 

3. What kind of training, in the form of a tutorial, would best prepare users to 
efficiently and effectively access the information system’s full 
capabilities? 

C. PURPOSE 

The intent of this study is to refine and evaluate the existing safety information 
management system, which uses the HFACS-ME taxonomy as its basis, to better 
facilitate data collection, organization, query, analysis and reporting of maintenance 
personnel errors that contribute to Naval Aviation mishaps, equipment damage and 
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personnel injury. Included are theoretical approaches that examine mishaps involving 
human error: Heinrich’s “Domino” Theory, Edwards’ “SHEL” Model, and Reason’s 
“Swiss Cheese” Model. These models not only help identify the unsafe actions which 
cause mishaps, but they also connect the conditions underlying the mishap. HFACS-ME 
is a composite derivative of these taxonomies. Although there is no universally accepted 
method of accident investigation (Benner, 1975), some standardized aircraft accident 
investigation procedures have been adopted by most agencies throughout the world 
(Diehl, 1991). Using HFACS-ME as its basis, MEEMS will significantly improve access 
and analysis of mishap data. 

The results of this study will (1) improve and enhance the MEEMS prototype 
tool’s usability so that data can be easily queried and analyzed, (2) creation of a tutorial 
for fleet customers that will enhance their ability to not only access the data base, but to 
efficiently and effectively use this information to prevent future mishaps, and (3) study 
the performance of two groups of participants, with only one group receiving the tutorial, 
to determine the effectiveness of the tutorial for fleet users. 

D. SCOPE AND LIMITATIONS 

MEIMS will be revised to include a users’ tutorial. Fleet personnel, primarily 
Aviation Safety Officers, will test MEIMS to determine its usability and the benefits of 
the help and tutorial functionality. The fully developed MEEMS tool will be used 
primarily by Naval Aviation squadrons, but may have some crossover use by other 
military branches and civilian airlines. Only maintenance related mishaps caused by 
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human error are considered. No material failure factors or maintenance related hazard 



reports or personnel injuries not related to a mishap are included. 

E. DEFINITIONS 

This study uses the following definitions: 

Aircraft Mishap Board : Group of officers appointed to investigate and report on 
an aviation mishap. 

Aviation mishap rate : Number of aviation mishaps per 100,000 flight hours. 
Aviation Safety Officer : Principal advisor to Naval Aviation squadron 
commanding officers on all aviation safety matters. 

F-14 Tomcat : U. S. Navy’s swing-wing, supersonic fighter aircraft with air-to-air, 
air-to-ground, and reconnaissance capability. 

Fleet Logistics Support Wing : U. S. Navy reserve air wing comprised of transport 
aircraft. 

HFACS : Human Factors Analysis and Classification System designed to help 
analyze Naval Aviation mishaps focusing on aircrew error. 

HFACS -ME : Human Factors Analysis and Classification System— Maintenance 
Extension adaptation to classify causal factors that contribute to maintenance mishaps. 

HFQMB : Human Factors Quality Management Board established by Naval 
Aviation senior leadership to reduce human error involvement in Naval Aviation Class A 
flight mishaps. 

MEIMS : Maintenance Error Information Management System, a prototype tool 
developed to collect, catalog, collate and analyze mishap data. 
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Mishap : A naval mishap is an unplanned event or series of events directly 
involving naval aircraft, which result in $10,000 or greater cumulative damage to naval 
aircraft, other aircraft, property, or personnel injury. 

Mishap Categories: Naval aircraft mishap categories are defined below: 

Flight Mishap (FM) : Those mishaps in which there was $10,000 or 
greater DOD aircraft damage or loss of a DOD aircraft, and intent for flight for 
DOD aircraft existed at the time of the mishap. Other property damage, injury, or 
death may or may not have occurred. 

Flight Related Mishap (FRM) : Those mishaps in which there was less 
than $10,000 DOD aircraft damage, and intent for flight (for DOD aircraft) 
existed at the time of the mishap, and $10,000 or more total damage or a defined 
injury or death occurred. 

Aircraft Ground Mishap (AGM) : Those mishaps in which no intent for 
flight existed at the time of the mishap and DOD aircraft loss, or $10,000 or more 
aircraft damage, and/or property damage, or a defined injury or death occurred. 
Mishap Severity Class : Mishap severity based on injury and property damage. 

Class A : A mishap in which the total cost of property damage 
(including all aircraft damage) is $1,000,000 or greater; or a naval aircraft is 
destroyed or missing; or any fatality or permanent total disability occurs with 
direct involvement of naval aircraft. 

Class B : A mishap in which the total cost of property damage 
(including all aircraft damage) is $200,000 or more, but less than $1,000,000 
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and/or a permanent partial disability, and/or the hospitalization of five or more 
personnel. 

Class C : A mishap in which the total cost of property damage 
(including all aircraft damage) is $10,000 or more but less then $200,000 and/or 
injury results in five or more lost workdays. 

Naval Aircraft : Refers to U.S. Navy, Naval Reserve, and Marine Corps aircraft. 
OPNAVINST 3750.6: The Naval Aviation Safety Program : U.$. Navy instruction 
outlining Naval Aviation’s safety program. 

ORM: Operational Risk Management. A decision making tool to increase 
effectiveness by anticipating and reducing hazards, thus increasing the probability of a 
successful mission. 

F. ORGANIZATION OF STUDY 

Chapter II contains a literature review on the development of a prototype to 
identify human error involvement and patterns in aviation maintenance mishaps. The 
methods used in this study are discussed in Chapter III. The results of this study are 
presented in Chapter IV. Finally, Chapter V contains conclusions, findings, and 
recommendations. 
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II. LITERATURE REVIEW 



A. OVERVIEW 

The literature studied encompasses human error, maintenance error in aviation, 
error classification and analysis. It includes textbooks, research articles, and masters 
thesis pertaining to: (1) human error theories, and their relation to maintenance related 
aviation mishaps, (2) accident information management, and (3) design and usability 
assessment of software tools. This information provides the basis for the continuing 
development of the maintenance error analysis and reporting prototype tool. 

While ongoing efforts to reduce the number of Class A mishaps in Naval Aviation are to 
be applauded, there is potential for further improvement, especially in the area of human 
error. When examining mishaps involving human error there are numerous theoretical 
approaches from which to choose (Goetsch, 1996). Some are founded in industrial 
safety; others are viewed from a complex systems perspective, emphasizing human 
factors and operator error. Some approaches use models drawn from the domain of 
preventive medicine that employ epidemiological factors to analyze accidents, while 
other models utilize a combination of these approaches (Schmidt, 1998). 

B. ACCIDENT DATA AND MANAGEMENT 

1. Investigation and Reporting 

Diehl’s (1991) three-stage model of accident investigation and prevention focuses 
on human performance and systems safety considerations (see Figure 2). Diehl’s stages 
are Accident Generation, Accident Investigation Process, and Preventive Measures. The 
purpose of the first stage. Accident Generation, is the identification of hazards. Hazards 
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are circumstances or conditions which have the potential to lead to an incident (near- 
accident) or even an accident. The second stage. Accident Investigation Process, 
includes the collection, analysis, and reviev/ of accident data and the focus of this review. 
In aircraft accident investigations, procedure calls for a fact finding body to investigate 
the accident to determine what happened, and subsequently what caused accidents. 
Lastly, a final report is made, which details causes of accidents and makes 
recommendations to prevent reoccurrence (Diehl, 1991). 




ACCIDENT PREVENTION 



Figure 2: Accident Generation, Investigation, and Prevention Elements 

(After Diehl, 1989) 



The final stage. Prevention Measures, details the methods used to avoid future 
accidents. There are four categories of accident-prevention measures: (1) eliminating 
hazards and risks, (2) incorporating safety features (3) providing warning devices, and 
(4) establishing procedural safeguards. In ascending order, these measures are more 
effective, but are also more expensive and more difficult to attain (Diehl, 1991). 
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2. Accident Prevention 



In the early 1900s, accident investigations were based on the notion people 
commit unsafe acts that lead to incidents (Heinrich, 1959). This blame-based practice 
deterred systematic accident prevention until the 1950’s, when systems engineering 
emerged from of the U.S. military’s large-scale weapons program. Applied to accident 
prevention, systems engineering focuses on the strengths and limitations of components, 
including the human element, as an integral part of the system (Heinrich, Petersen & 
Roos, 1980). Presently, up to 90 percent of accidents can be attributable to human error 
in a variety of government, military and industrial settings (Heinrich, Petersen, & Roos, 
1980; Hale & Glendon, 1987; School of Aviation Safety, 2000), so analyzing the human 
element can vastly improve the understanding of systems and why they fail. 

Operational Risk Management (ORM), an outgrowth of systems engineering, is a 
tool that the armed forces employed to decrease accidents. ORM focuses on the 
identification of hazards during mission planning to effect control measures which reduce 
the associated risk. It is especially effective in highlighting human factor hazards (DON, 
1997). The U.S. Army's aviation branch achieved record low accident rates in 1995 and 
1996, largely attributable to their employment of ORM (DOA, 2000). Since 1997, the 
other military services institutionalized ORM into their operational doctrine (School of 
Avaition Safety, 2000). 

3. Data Management 

To be useful in the prevention of accidents, relevant data must be collected and 
properly archived for future reference. Coding the data and the use of data bases have 
become universally accepted methods for this task. The National Safety Council 
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established a numerical code system relating to each cause factor (National Safety News, 
1975). This method of coding simultaneously made referencing for specific cause factors 
much simpler, and also provided more efficient analysis capability by focusing on 
specific causes rather than accidents as a whole. Additionally, it is essential to also 
include “near-miss” incident data, as their circumstances are as likely to cause an 
accident in the future. Computer analysis tools can significantly aid reviews of mishap 
histories (Grimaldi & Simonds, 1984). To be truly effective in reducing future accidents 
and incidents those tools must organize and tabulate the data logically, and present it for 
analysis in easily accessible, user friendly formats. 

C. ACCIDENT CAUSATION THEORIES 

There are numerous theoretical approaches to examine mishaps involving human 
error (Goetsch, 1996). Three of the most recognized approaches are Heinrich’s Domino 
Theory, Edwards’ SHEL Model and Reason’s Swiss Cheese Model. Despite differences 
in approach and originating source, these theories compliment each other and provide a 
solid foundation for analyzing human error in avaition mishaps. 

1. Heinrich’s “Domino” Theory 

Considered to be the original accident theory (Goetsch, 1996), it details a linear 
five step sequence of related factors (chain of events) that lead to an actual mishap. Bird 
(1980) also held this view. The two central principles of the Domino Theory are (Bird, 
1980): (1) accidents are caused by the combined actions of the preceding factors, and (2) 
removal of the middle factor (unsafe act or condition) will negate the actions of the 
preceding factors and thus prevent accidents and injuries (see Figure 3). 
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Figure 3: The Five-Step Domino Sequence (Bird, 1980) 

Heinrich’s original five-factor model (Heinrich, Petersen, & Roos, 1980), can be 
paraphrased as follows: 

1. Lack of Control: This is a management issue where the emphasis is placed 
on the control exercised in a situation for an array of factors. 

2. Basic Cause(s): This identifies the origin(s) of the causes and includes 
aspects such as human factors, environmental factors, or job-related 
factors. 

3. Immediate Cause(s): This includes substandard practices and conditions 
that are symptoms of the basic causes. 

4. Incident: This typically involves contact with the hazard, and for example, 
results in a fall or impact with moving objects. 

5. Personal Injury and Property Damage: This includes lacerations, fractures, 
death, and material damage. 

Each step preempts the next, causing it to occur; much the same way as one 
domino falling causes the next domino in sequence to fall as well. Removal of the 
factors that comprise any of the first three “dominos” will effectively intervene to prevent 
the accident. 
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2. Edwards’ “SHEL” Model 



The “SHEL Model” (see Figure 4) of system design was developed in the early 
1970’s to provide a better way to evaluate failures in human-machine systems (Hawkins, 
1992). It identifies and defines four system dimensions: Software, Hardware, 
Environment, and Liveware. The “SHEL Model” provides a method to describe systems, 
identify potential areas for concern within a system, and proposes a general framework 
for accident investigation. Edwards (1988) defines SHEL concepts as follows: 

Software: the rules, regulations, laws, orders, standard operating procedures, customs, 
practices, and habits that govern the manner in which the system operates and in which 
the information within it is organized. Software is typically a collection of documents. 

1. Hardware: the buildings, vehicles, equipment, and materials of which the 
system is comprised. 

2. Environmental conditions: the physical, economic, political and social 
factors within which the software, hardware, and liveware operate. 

3. Liveware: the human beings involved with the system. 

According to the “SHEL Model,” the system will fail when a disconnect occurs in 
any one of the four dimensions, or in the connections between them. People are rarely 
the sole cause of accidents (Edwards, 1988), but rather, accidents are caused by the 
interaction of several factors (Shappell & Wiegmann, 1997). This multi-factor theory 
was a significant departure from the commonly held belief that accidents were caused by 
single events. 
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3. Reason’s “Swiss Cheese” Model 

Reason’s “Swiss Cheese” Model employs a vertical association human factors 
approach to a collection of factors that eventually lead to an accident. It differentiates 
between two types of errors: 1) active failures, whose effects are felt immediately, and 
2) latent conditions, whose effects may lie dormant until triggered later, usually by other 
mitigating factors (Reason, 1990). Latent conditions “set the stage” for an accident while 
active failures tend to be the catalyst for the accident to finally occur. The model can be 
thought of as slices of Swiss cheese lined up, with each vertical slice representing a 
defense layer (e.g. training, good management, teamwork, etc.) and each hole 
representing an active failure or latent condition in that defense (see Figure 5). Should a 
situation where holes line up come to pass, an accident will occur. 

This is a dynamic model with each defensive layer coming in and out of 
prominence according to situational characteristics (Reason, 1990). An event may occur 
in one of three levels: (1) person-unsafe acts, (2) workplace-error provoking conditions, 
and (3) organization-error establishing conditions. Organizational factors, in which 
strategic decisions and associated processes (budgeting, forecasting, resource allocation) 
are initiated, provide the starting point for an accident (Reason, 1997). These processes. 
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influenced by corporate culture, are distributed throughout the organization to individual 
workplaces. Corporate processes, evidenced as inadequate staffing, time pressures, 
equipment, training and working conditions, combine with the natural proclivity to 
commit errors and/or violations, to result in unsafe acts. Few of these acts, however, 
actually create holes in the defense layers en route to becoming an accident. 




Figure 5: Reason’s Swiss Cheese Model (Naval Safety Center, 1996) 

D. HUMAN FACTORS ANALYSIS & CLASSIFICATION SYSTEM (HFACS) 

Efforts to analyze aircrew error in Naval Aviation mishaps led to the development 
of HFACS (Shappell & Wiegmann. 1997). It began as a restructuring and expansion of 
the Swiss Cheese Model, but it also includes features of Heinrich’s Domino Theory and 
Edwards’ SHEL Model. The HFACS taxonomy recognizes the set of conditions within 
which system operators perform, accounts for interrelationships among conditions, and 
provides an overall sense of order or implied hierarchy. It identifies both active failures 
and latent conditions within four categories (DON, 2000): (1) unsafe acts; (2) pre- 
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conditions for unsafe acts; (3) unsafe supervision, and (4) organizational influences. This 
classification is then used to target the most appropriate intervention. 

While Reason’s “Swiss Cheese” model is widely held as integral to explaining 
accidents, it does not provide a means of delineating precursors to accidents (Shappell &, 
Weigmann 1997). Although modem analysis systems are heading in the right direction 
regarding human factors investigation, more effort is needed. There exists a requirement 
for a framework that allows for the explanation of accident causation involving human 
error to include not only the causes of those errors, but also an explanation of those 
causes. It is through such efforts that further occurrences of the same type of accident 
can be avoided. 

1. Maintenance Mishaps 

The aviation industry has, in recent years, begun to apply human error theory to 
maintenance-related mishaps with the intent of modernizing accident investigations 
(Marx, 1998). Previously, it was believed that every error could be traced back to a basic 
set of actions and associated conditions that precipitated the error (Goetsch, 1996). This 
simplistic view did not allow for errors to have multiple causes, as most do. 

Additionally, traditional investigation techniques, while appropriate for identifying the 
causes of equipment failures, did not have the same success with human error-related 
accidents (O’Connor & Bacchi, 1997). Typically, investigations would effectively end 
when the cause pointed to human error, with no effort expended in attempting to explain 
why the error occurred. 

In a review of investigation and analysis systems for aircraft maintenance error, 
the need for human factors investigation and reporting was identified (Marx, 1998). 
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Although human factors investigation methods are acknowledged as critical to 
understanding why people make certain mistakes, they have not been widely adopted. 
The proper investigation of human factors, however, is vital to accident prevention 
(Harle, 1994). The lack of thorough human factors investigation was caused by the 
tendency to place blame, inability to see through proximate causes to underlying causes, 
and an over-emphasis on static factors such as who, what and when (Marx, 1998). 

2. HFACS-ME Maintenance Extension 

In light of the above requirements, the HFACS taxonomy was adapted, and 
augmented to classify factors that lead to maintenance mishaps (Schmidt, 1996). The 
Maintenance Extension (ME) of the original HFACS taxonomy focuses on causation 
factors particular to the maintenance environment. These aspects and their relevant sub- 
section classifications are given in a manner similar to Reason’s model. In the HFACS- 
ME model, a maintainer’s performance is seen as being influenced by a series of latent 
conditions (supervisory, maintainer and working conditions) that can lead to an Unsafe 
Maintainer Act, which in turn can lead to a mishap, injury or an unsafe maintenance 
condition (see Figure 6). 
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Figure 6: HFACS Maintenance Extension (HFACS-ME) 

(DON, 2000) 



HFACS-ME consists of four broad human error categories, three latent: 
Management Conditions (e.g. inadequate supervision), Maintainer Conditions (e.g. 
preparation/training). Working Conditions (e.g. lighting/light), and the fourth Maintainer 
Acts (e.g. skill error), is active (DON, 2000). The three orders of maintenance error 
(first, second, and third) reflect a decomposition of the error types from a macro to a 
micro perspective. Each successive order provides for greater granularity, serving the 
respective purposes of identifying problem areas, prioritizing potential targets, and 
tailoring intervention strategies (see Table 1). 
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Table 1: HFACS Maintenance Extension Categories (DON, 2000) 



First Order 


Second Order 


Third Order 


Management Conditions 


Organizational 


Inadequate Processes 
Inadequate Documentation 
Inadequate Design 
Inadequate Resources 


Supervisory 


Inadequate Supervision 
Inappropriate Operations 
Uncorrected Problem 
Supervisory Misconduct 


Maintainer Conditions 


Medical 


Adverse Mental State 
Adverse Physical State 
Unsafe Limitation 


Crew Coordination 


Inadequate Communication 
Inadequate Assertiveness 
Inadequate Adaptability/Flexibility 


Readiness 


Inadequate Training/Preparation 
Inadequate Certification/Qualification 
Personnel Readiness Infringement 


Working Conditions 


Environment 


Inadequate Lighting/Light 
Unsafe Weather/Exposure 
Unsafe Environmental Hazards 


Equipment 


Damaged/Unser viced 
Unavailable/Inappropriate 
Dated/U ncertified 


Workspace 


Confining 

Obstructed 

Inaccessible 


Maintainer Acts 


Error 


Attention/Memory 

Knowledge/Rule 

Skill/Technique 

Judgment/Decision 


Violation 


Routine 

Infraction 

Exceptional 

Flagrant 



3. Maintenance Error Information Management System (MEIMS) 

The HFACS-ME taxonomy was used as the framework for the development of a 
laptop prototype database tool, MEIMS. It is intended for fleet users to collect, catalog, 
collate and analyze mishap data, and to better identify trends and for safety training. In 
limited usability testing, MEIMS was proven to be effective, yet was lacking in some 
areas. Specifically MEIMS was somewhat difficult to navigate, was lacking in assistance 
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functionality and it’s customer interface needed to be more user-friendly. The benefit of 
user training was also indicated. To fully reach its potential, however, MEIMS needs 
design refinements and more rigorous usability testing (Wood, 2000). 

E. USER INTERFACE 

The proliferation of computer systems into ever increasing environments 
highlights the need for effective and easy to use interfaces. Clearly, the user interface is 
the most important factor in determining the success or failure of a software application 
(Liu, 1997). The design objectives of user interface include effectiveness, efficiency, 
comfort and safety. The content of an interface should abstract critical features of the 
target system (Norman, 1983; Vicente & Rasmussen, 1992), be consistent with human 
conceptual and cognitive characteristics (Carroll, Mack & Kellog, 1988; Wickens, 1992), 
be compatible with human response tendencies (Myers, 1991; Shneiderman, 1983), and 
should help users effectively achieve their intended impact on the target system 
(Sheridan, 1984; Shneiderman, 1992). 

Williges, Williges & Elkerton (1987) examined design objectives from user's 
perspective and generated a list of seven general design principles as a basis for 
developing specific design objectives. The principles are (1) consistency, (2) 
compatibility, (3) memory, (4) structure, (5) feedback, (6) workload, and (7) 
individualization. According to these principles, a well designed user interface should 
maintain the same style of interaction within itself and similar applications (consistency), 
provide easily understandable interface symbols and icons to minimize the demand for 
information recoding (compatibility), minimize the demand on the user's short-term 
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memory (memory), help users understand the structure of the system (structure), provide 
feedback and error-correction mechanisms (workload), and accommodate individual 
differences (individualization). 

From an engineering point of view, Shneiderman (1992) detailed four essential 
classes of goals for interface design. They are (1) functionality to accomplish all 
assigned tasks, (2) foolproof reliability, availability, security and data integrity, (3) 
standardized styles and interface features to ensure portability over multiple applications, 
and (4) development should maintain schedule and remain within budget. The goal of 
interface design not only includes assisting the user in accomplishing their task of 
controlling or influencing a target system, it should also help the user understand the 
system. 

In most cases, the target system and the user interface are usually designed by 
separate individuals, who are often from different groups or organizations. The design 
activities may take place at different places and at different times, making effective 
communications between the two groups especially challenging. The two design teams 
may have different objectives, constraints and conceptual bases. Interface designers must 
establish early contact with not only future users of the system, but also designers of the 
system so that both the "user model" and the "design model" can be clearly identified and 
correctly reconciled. Interface designers must recognize and overcome these competing 
models to successfully bridge the gap between the user’s mental model and the system 
designer’s conceptual model of the same target system (Norman, 1983). 
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F. USABILITY TESTING 



Although user testing with real users is the most fundamental usability method, 
other usability methods can serve as good supplements to gather additional information 
or gain usability insight at a lower cost. User testing, combined with heuristic evaluation 
and other usability inspection methods, can provide better efficiency (Nielsen, 1997). 
Heuristic evaluation consists of finding usability problems in a design by contrasting it 
with a list of established usability principles. Though this method works very well and 
produces fast evaluation results, there are always some issues that will be left 
undiscovered until actual user testing (Nielsen, 1997). In all testing, issues of reliability 
and validity need serious consideration. Reliability considers whether one would get the 
same result if a test were repeated and validity considers whether the result actually 
reflects the usability issues one wants to test. 

1. Reliability 

Huge individual differences between test users presents a reliability problem with 
usability testing. It is not uncommon for the best user to be 10 times faster than the 
slowest user, or the top 25 percent of the users to be twice as fast as the slowest 25 
percent of the users (Egan, 1988). Due to this phenomenon, one cannot conclude much 
from observing a user A using interface X performing a certain task 40 percent faster 
than user B using interface Y. User B could very well just happen to be slower than user 
A. If the test were repeated with users C and D, the result could easily be exactly 
opposite. Often decisions must be made on the basis of fairly unreliable data, which 
while suspect, is better than having no data at all. Standard statistical tests should be used 
assess test results and thereby indicate the significance and reliability. Although a 0.05 
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significance level is often used for research studies, for practical development purposes 
an 0.20 significance level might be more attainable due to budget and time restraints 
(Nielsen, 1997). 

2. Validity 

While reliability can be addressed with statistical tests, a high level of validity 
requires full understanding of the test method, as well as common sense. Typical validity 
problems include using the wrong users, giving them the wrong tasks, or not including 
time restraints of social influences. Confounding effects may also lower the validity of a 
usability test. An example would be testing the move from a character based user 
interface to a graphical user interface for an application. If, while running this test, the 
two competing systems were running on screens set to different pixel graphic displays, 
the test results may give a comparison between the two size screens rather than the 
intended character-based versus graphical user interfaces (Nielsen, 1997). 

3. Test Users 

The primary concern regarding test users is that they should be as representative 
as possible of the intended users of the system. Sometimes the exact individuals who will 
be using a system can be identified. Typically, this case is specific to a system being 
developed for an organization or department within an organization. Another case is a 
system targeted at a certain type of user, such as lawyers, secretaries or warehouse 
managers. In this case the users can be more or less homogeneous, but it is desirable to 
select users from as many different locations as possible. For software intended for the 
general population, almost anyone can serve as a test user. Lastly, care must also be 
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taken to ensure that both expert, novice, and mid skill level users are chosen (Neilsen, 
1997). 

4. Test Tasks 

Test tasks should be as representative as possible of the uses to which the system 
will eventually be fielded. They should also reasonably cover the most important parts of 
the user interface, be small enough to be completed within the test time limits, yet not 
become trivial. The first task should be extremely simple to guarantee the user an early 
success experience to boost morale. Conversely, the last task should provide the user 
with the feeling of accomplishment (Gaylin, 1986). 

Usability tests typically have four stages (Nielsen, 1997): 

1. Preparation: Ensure the test room is ready and that the computer system is 
in the start state, and that all test materials and instructions are available. 

2. Introduction: Welcome the test user and give a brief expalnation of the 
purpose of the test, to include computer setup if necessary. 

3. Running the test: During the test itself, the experimenter should normally 
refrain from interacting with the user. The exception to this rule is the 
case where the test users is clearly stuck and becoming quite unhappy with 
the situation. Caution must be exercised not to help the user too early. 

4. Debriefing: After the test, debrief the user and get any subjective input as 
soon as possible. During the debrief, extract from the user their input 
regarding the test itself. Also, as soon as possible, ensure that all results of 
the test have been collected and properly labeled. 

Consideration of the required tasks is critical to properly setting up the test. It is 
important that the test user feel as if their input is significant, beneficial and appreciated. 
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5. Performance Measurement 



Measurement studies for the basis of much traditional research on human factors. 
User performance is almost always measured by having a group of test users perform a 
predefined set of test tasks while collecting time and error data (Nielsen, 1997). There is 
the potential, however, to measure something that is poorly related to the property which 
is attempting to be assessed. Goals are generally abstract, so care must be taken to break 
them down into components with specific usability attributes, which are to be measured 
by specific tasks. Some usability studies use qualitative methods rather than exact 
measurements. The focus of the test could be to determine which aspects of the system 
work well and which are troublesome, yet not have any exact measures to differentiate 
the two. To mine all possible observations of qualitative data analysis requires extensive 
engineering experience. Even inexperienced usability engineers, however, can usually 
ascertain major qualitative findings, as they are more likely to be readily apparent 
(Nielsen, 1997). 

G. SUMMARY 

Naval Aviation has made significant strides in reducing the mishap rate in the last 
half century. These advances can be attributed to more standardized investigation and 
reporting systems, the use of systems engineering and the application of human error 
causation theory on mishap cause factors. Significant amounts of data are being 
collected, cataloged and stored for future reference. Computer database systems make 
the organization of this data relatively simple. For Naval Aviation to take the next step, 
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however, a robust system for analyzing stored data for trend analysis and prevention is 
needed. 

In an effort to provide a more useful data analysis system, HFACS was 
developed, then expanded to cover MRMs in HFACS-ME. Proven to be effective in 
capturing the nature of, and relationships among, latent conditions and active failures. 

The HFACS-ME taxonomy was developed into a laptop prototype database tool, 

MEIMS. In a limited usability study, MEEMS was discovered to have great potential. To 
fully attain its potential, however, MEIMS must be proven to be a user friendly system 
which fleet users will embrace. Further system development and more rigorous usability 
testing is required. With proper advances, MEIMS can be especially beneficial in 
reducing aviation maintenance related mishaps. 
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III. METHODOLOGY 



A. RESEARCH APPROACH 

A desktop investigating, reporting and analysis system focusing on maintenance 
error in aviation mishaps can greatly advance efforts to enhance safety and reduce 
mishaps. This information management system’s database is populated with historical 
records of aviation maintenance mishap information. Additionally, a cursory training 
lesson and a hands-on experience tool are developed which assists the participants in the 
use of the database. Next, the prototype version of the tool is distributed to two sample 
groups; one getting the training lesson and an operator's manual and the others getting the 
training session, an operator's manual, and a hands-on experience session. 

Participants in both groups are then provided a prepared task list that required 
them to navigate through the system. At the completion of the task list, all participants 
have viewed all portions of the prototype system, and have had the opportunity to form 
an opinion on its effectiveness. The two groups’ performance is evaluated to determine 
the usefulness of the hands-on experience session in completing the assigned tasks. 
Additionally, all participants complete an exit survey composed of demographic 
background questions, opinion items of the prototype system (including improvements), 
opinion items on the value of the training lesson and the hands-on experience session. 
They also answer open-ended questions on the prototype system, training session and 
hands-on experience session. 
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B. DATA COLLECTION 



1. Participants 

Students attending the Naval Postgraduate School (Monterey, CA) serve as 
participants in the study. Most of the participants have recently come from fleet 
department head tours or embarked airwing staffs. These aviators are from all aircraft 
communities and within the U.S. Navy and Marine Corps. All have had some experience 
with mishap information reports and the data contained therein. They are representative 
of the type of people that will likely be the primary users of the tool. The participants 
were split into two groups of ten. The groups were distributed evenly matching rank, 
branch of service, type aircraft flown and years of aviation experience. One group 
receives a training lesson on the tool along with the instruction manual and the other 
group gets the training lesson, instruction manual and an additional in depth tutorial. 

2. Apparatus 

The participants are introduced to the Human Factors Analysis and Classification 
System-Maintenance Extension (HFACS-ME) taxonomy and to the Maintenance Error 
Information Management System (MEIMS) prototype data base tool. Additionally, the 
students have access to three computer labs at the School of Aviation Safety via login ID 
and password to a group account. Each computer in the laboratories is a Pentium I, 
Windows 2000 operating system, with 15-inch monitor of 800 x 600 resolution or better. 
Each has a full functioning prototype of the tool loaded onto it. Prior to logging on, both 
groups are given a cursory training lesson and instruction manual in the use of MEIMS. 
Additionally, one group is provided with an additional in depth tutorial in HFACS-ME 
taxonomy and MEIMS capabilities. After a participant gains access to the computer, the 
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icon "MEIMS Tool” is found on the computer desktop. Once the icon is selected, the 
prototype opens using Microsoft Access 2000. 

3. Instrument 

The participant completes a usability test consisting of eleven tasks. After 
completion of the test, an exit survey is constructed for completion by the participants 
consisting of three parts: (1) demographics of the participant, (2) Likert-type questions 
assessing feelings towards the prototype tool (including improvements), the hands-on 
experience session and tutorial (as applicable), and (3) open-ended items to elicit 
subjective responses. Participants receiving the tutorial answer additional Likert-type 
questions and open-ended questions regarding the tutorial. Survey questions are designed 
to determine if the tool meets user investigation, reporting, and analysis requirements. 

The survey also queries the participants getting the tutorial in its value. Additionally, 
participant performance between the two groups is evaluated to determine the 
effectiveness of the tutorial. 

Collection of demographic information is accomplished through the participant 
selecting from a list of descriptors (squadron, experience with computers, software 
familiarity, and operating system knowledge). The Likert-type questions use a five point 
rating scale with verbal anchors: Strongly Agree, Agree, Neutral, Disagree, and Strongly 
Disagree. Open-ended questions are included to gain inputs on the overall impression of 
the training lesson, prototype tool, the hands-on experience session, the tutorial (as 
applicable), and recommendations for improvement. 
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4. Procedure 



MEIMS was revised and updated to include data derived from an available 
database; a sample group of users will measure its effectiveness. The participants are 
split into two similar groups. One group is provided an in depth tutorial (See Appendix 
B) prior to using MEIMS. The perceptions and performance of both groups are measured 
to determine the value of the MEIMS and the tutorial. 

All Participants are given an overview presentation of the study consisting of a 
projected computer demonstration of the prototype tool and distribution of materials 
necessary to carry out the user test. The overview consists of: 

• Instructions for Accessing the Prototype Tool - information to log on and open the 
prototype. 

• MEIMS Evaluation (Appendix C)— a series of planned navigation routes within 
the prototype whereby the participant would be able to view the entire system. 

• MEIMS Exit Survey (Appendix C)~ Participants complete an exit survey 
composed of demographic background questions, impressions of MEIMS and the 
tutorial (as applicable) and open-ended question requesting participant’s opinions. 
All participants perform the above actions: access the prototype tool, navigate the 

system using the prototype task list, and provide feedback on the system by completing 
the exit survey. Additionally, the group receiving the tutorial makes comments on its 
effectiveness and usefulness. The performance of the tutorial group is compared with the 
non-tutorial group to determine the benefits of the tutorial. 
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C. DATA TABULATION 



The data is transcribed from the survey onto a Microsoft Excel 2000 spreadsheet. 
The Likert-type questions, based on a five-point scale, are coded into the software (1 
through 5) corresponding to the respective anchors (Strongly Disagree, Disagree, Neutral, 
Agree, and Strongly Agree). Specific categories (MEIMS is: Logical, Easy to Navigate, 
Very Interesting, Relevant and a Good Concept) are examined for significant differences 
between the tutorial and non-tutorial groups. Open-ended questions are tabulated by 
theme and number of responses. 

D. DATA ANALYSIS 

The data analysis tools of Microsoft Excel 2000 are used to generate descriptive 
statistics including the mean, standard deviation, and range of the collected data. The 
performance of the two groups is tabulated, measured and compared. Independent t-tests 
are conducted on the data to determine significant differences between the tutorial and 
non-tutorial groups. Participant responses to the open-ended questions are also explored 
to assess the perceived value of the hands-on experience session, the MEIMS tool itself 
and the tutorial. 
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IV. RESULTS 



A. TEST SAMPLE 

A usability test was administered to twenty students at the Naval Postgraduate 
School. They were randomly split into two groups of ten each. Many of the participants 
(n=17; 85%) were designated Naval Aviators or Naval Flight Officers and represented a 
cross section of the aviation commands that make up the squadrons in the Navy and 
Marine Corps. One group of ten received a formal tutorial prior to testing the MEIMS 
prototype, while the other did not. One of the non-aviators was in the tutorial group, two 
were not. Both groups were directed to access MEIMS and complete an eleven-item task 
list. Following the tasks, individual participants were asked to complete an exit survey. 
The survey consisted of demographic information, queries regarding their satisfaction 
with MEIMS and queries regarding their satisfaction with the tutorial, if applicable. 

B. TEST TASKS 

The tasks portion of the test was designed to not only introduce the participants to 
MEIMS and exercise some of its capabilities, but also to test the effectiveness of the 
tutorial. The tasks required the participant to access all functional areas of MEIMS. The 
tasks started relatively simply but became more complex as the participant became more 
familiar with MEIMS. Test performance is summarized in Table 2. 
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Table 2: Test Task Performance 





NUMBER OF CORRECT RESPONSES 


TASK 


TUTORIAL 


NON-TUTORIAL 


1 


10 


10 


2 


10 


10 


3 


10 


10 


4 


10 


10 


5 


10 


10 


6 


10 


7 


7 


10 


10 


8 


9 


6 


9 


10 


10 


10 


10 


10 


11 


10 


10 



The first task was accessing the program; all participants (n=20; 100%) were able 
to access MEIMS with no difficulty. The second task requested the participant’s opinion 
of the main menu. A number of participants from both the tutorial group (n=6; 60%) and 
the non-tutorial group (n=4; 40%) noted that the tab order was not sequencing properly in 
the main menu. The third task required the participant to query a type of aircraft and 
determine how many mishaps exist in the database for that type aircraft. Secondly, the 
participant had to identify and define the level 3 codes. All participants (n=20; 100%) 
were able to correctly complete this task. The fourth task asked the participant which of 
the seven categories was most useful and which was least useful. Aircraft model (n=17; 
85%) was considered the most useful, while Location of Mishap and Aircraft Type (n=6; 
30%) were each viewed as the least useful. 

The fifth task asked the participants to select their own criteria for a multiple 
query. Some participants (n=5; 25%) created queries that were too restrictive for the 



36 



limited number of mishaps in the prototype database. Once they viewed the error 
message that their query was too restrictive, they were able to reselect and complete the 
query. Both the tutorial (n=8; 80%) group and non-tutorial (n=7; 70%) group thought 
this feature was extremely useful. The sixth task required the participant to access the 
HFACS-ME Summary query and determine how many mishaps were in the database. 

All participants (n=20; 100%) correctly obtained this information. The participant then 
had to determine how many level one Worker Condition mishaps were in the database 
and how many level two Medical conditions were in the database. All of the tutorial 
group (n=10; 100%) correctly answered this question, however not everyone in the non- 
tutorial group (n=7; 70%) correctly answered it. The seventh task asked the participant to 
determine the number of mishaps of another type aircraft. All participants (n=20; 100%) 
correctly determined the number of mishaps. 

The eighth task required the participant to determine the number of Detached 
mishaps in the Mishap Distribution - Location Report. In the tutorial group, almost all 
(N=9; 90%) correctly answered this question. In the non-tutorial group, over half (n=6; 
60%) correctly answered it. The ninth task asked the participant to graph two type 
aircraft on the X-Axis versus HFACS-Level One codes on the Y-Axis. All participants 
(n=20; 100%) were able make the graph. Some participants in both the tutorial (n=4; 
40%) and the non-tutorial (n=2; 20%) groups noted that the graph provided frequency of 
mishaps, vice some normalized baseline, like percent of mishaps per type aircraft. The 
tenth task was to input two mishaps into the database. All members of the both groups 
(n=20; 100%) were able to successfully enter mishaps into the database. It was noted by 
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many participants in both groups (n=16; 80%) that the database does not adequately error 
check the information being input. 

The eleventh task asked the participant to check the Chronological Listing of the 
mishaps that had been input into MEIMS. Again, all the participants (n=20; 100%) 
correctly completed this task. Overall the performance of the tutorial group was slightly 
better than the non-tutorial group (See Figure 7). In the tutorial group almost all (n=9; 
90%) had no errors, while in the non-tutorial group less than half (n=4, 40%) had no 
errors. 




0 0 0 0 

Number of Errors 



□ Tutorial Group □ Non-Tutorial Group 

Figure 7: Test Task Performance 



The mean percentage of correct responses for the tutorial group was 99. 1 (SD = 
2.84, RNG = 10). whereas the non-tutorial group mean was 93.7 (SD = 6.07, RNG = 19). 
An independent teq Ua i test comparing mean tutorial and non-tutorial group performance 
was conducted. It assumed equal n, no difference between group means, and 
homogeneity of variance. The observed t^ai of 2.55 was significant at the p = 0.02 level 
for a two tailed test (See table Al, Appendix A). Given the variance for the tutorial 
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group was 8.1 and for the non-tutorial group was 36.9, a second t-test was run for 
unequal variance to reduce the chance of Type I error. The results are presented in Table 
3. The observed t une qua] of 2.55 was significant at the 0.02 level. 

Table 3: Task Performance Two-Sample t-Test Assuming Unequal Variance 





T utorial 


Non-tutorial 


n 


10 


10 


Mean 


99.10 


93.70 


Variance 


8.10 


36.90 


df 


13 




tCrit 


2.16 


tStat 


2.55 


P two-tail 


0.02 



C. DEMOGRAPHIC INFORMATION 

Part I of the exit survey requested demographic information from the participants. 
It established computer experience levels to determine if computer or aviation experience 
had an impact on that participant’s ability to utilize, or their satisfaction with, the MEIMS 
tool. Demographic information is summarized in Table 4. 
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Table 4: Demographic Information 





Number of Participants 


Demographic 


Tutorial 


Non-Tutorial 


SQDRN MX 


9 


7 


2 + YRS Computer EXP 


10 


10 


Microsoft Office 


10 


10 


Lotus Smart Suite 


1 


1 


Corel Office 


0 


1 


Word Processing 


10 


10 


Presentations 


10 


10 


Graphic Software 


10 


10 


E-Mail 


10 


10 


Database 


4 


6 


Windows (3.1-2000) 


10 


10 


Windows NT 


10 


10 


Macintosh 


1 


0 


Linux 


0 


1 



Question one determined that 9 of the tutorial group participants had been 
members of aviation units that performed squadron level maintenance (n=9; 90%). The 
other member of the tutorial group was from a non-aviation background (n=l; 10%). In 
the non-tutorial group, 7 participants were members of aviation units that performed 
squadron level maintenance (n=7, 70%), one was a member of a staff that did not 
perform maintenance (n=l; 10%), and two were from non-aviation backgrounds (n=2; 
20). Question two indicated that all participants had at least two years of experience 
using a computer (n=20; 100%). Question three determined that all participants (n = 20; 
100%) were users of Microsoft Office. Minimal numbers in either group had used either 
Lotus Notes (n = 2; 20%) or Corel/Word Perfect (n = 1; 10%). Question four established 
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a participant’s familiarity with different software applications; all participants (n=20; 
100%) were familiar with processing, spreadsheet, presentation, and e-mail. In the 
tutorial group, four were familiar with database applications (n=4; 40%), while in the 
non-tutorial group, that number was 6 (n=6; 60%). Question five determined that all 
participants (n=20; 100%) were familiar with Windows 97/2000, Windows NT operating 
systems, or both. The MEEMS tool was loaded on computers running the Windows NT 
operating systems. 

D. USER SATISFACTION WITH MEIMS TOOL 

1. Impressions of MEIMS 

Part II of the exit survey requested the participant’s impressions of the MEIMS 
tool and its value to Naval Aviation. Participants responded to five statements using 
Likert type responses selecting from one of five responses: strongly agree, agree, neutral, 
disagree, and strongly disagree. Values of five through one respectively were assigned to 
the statements. Additionally, participants could make subjective comments on any of the 
statements. 

Statement one asked whether or not a participant found MEIMS to be presented in 
a logical form. The histogram of the frequency distribution for statement one is 
presented in Figure 8. The mean for the tutorial group was 4.4 (SD = 0.52; RNG = 2), 
whereas the non-tutorial group mean was 4.1 (SD = 0.57; RNG = 3). All participants in 
the tutorial group agreed that MEIMS presented the information in a logical fashion 
(n=10; 100%), while in the non-tutorial group it was less than unanimous (n=9; 90%). 
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□ Tutorial □ Non-Tutorial 



Figure 8: MEIMS is in a Logical Form 



Statement two asked about the ease of navigation of the prototype. The histogram 
of the frequency distribution for statement two is presented in Figure 9. The mean for the 
tutorial group was 4.0 (SD = 0.5: RNG = 3), whereas the non-tutorial group mean was 
3.5 (SD = 0.97; RNG = 4). A majority of participants in both the tutorial group (n=9: 
90%) and the non-tutorial group (n=6: 60%) agreed that MEIMS was easy to navigate. 
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□ Tutorial Group □ Non-Tutorial Group 



Figure 9: MEIMS is Easy to Na\igate 
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Statement three asked if the participants felt MEIMS was “very interesting.” The 
histogram of the frequency distribution for statement three is presented in Figure 10. The 
mean for the tutorial group was 4.2 (SD = 0.63; RNG = 3), whereas the non-tutorial 
group mean was 3.7 (SD - 0.82; RNG = 4). Most participants in both the tutorial group 
(n=9; 90%) and the non-tutorial group (n=7; 70%) agreed that MEIMS was very 
interesting. 
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Figure 10: MEIMS is Very Interesting 



Statement four asked about the relevance of MEIMS to aviation maintenance 
operations. The histogram of the frequency distribution for statement four is presented in 
Figure 1 1. The mean for the tutorial group was 4.6 (SD = 0.52; RNG = 2), whereas the 
non-tutorial group mean was 4.5 (SD = 0.53; RNG = 2). All participants agreed that 
MEIMS was relevant (n=20; 100%). 
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□ Tutorial Group □ Non-Tutorial Group 



Figure 11: MEIMS is Relevant to Maintenance Operations 

Statement five asked whether prototype concept was a good one. The histogram 
of the frequency distribution for statement five is presented in Figure 12. The mean for 
the tutorial group was 4.6 (SD = 0.52: RNG = 2), whereas the non-tutorial group mean 
was 4.5 (SD = 0.53, RNG = 2). Again, all participants agreed that the concept of MEIMS 
was a good one (n=20; 100%). 
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□ Tutorial Group □ Non-Tutorial Group 



Figure 12: MEIMS Concept is Good 
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Independent t equaJ tests comparing tutorial and non-tutorial responses to the Likert- 
type questions were conducted. They assumed equal n, no difference between group 
means, and homogeneity of variance (See Appendix A). For the Logical, Navigation, 
Relevance and Concept categories, no significant difference was noted. For the 
Interesting category, an observed teq U ai of 2.59 was significant at the p = 0.02 level for the 
two tailed test (See table A4, Appendix A). Given the variance for the tutorial group was 
0.28 and for the non-tutorial group was 0.68, a second t-test was run for unequal variance 
to reduce the chance of Type I error. The results are presented in Table 5. The observed 
t unequal of 2.59 was significant at the 0.02 level. 

Table 5: MEIMS is Interesting Two-Sample t-Test Assuming 

Unequal Variance 





Tutorial 


Non-tutorial 


n 


10 


10 


Mean 


4.50 


3.70 


Variance 


0.28 


0.68 


df 


15 




tCrit 


2.13 


tStat 


2.59 


P two-tail 


0.02 



2. Open-ended Questions 

Part El of the exit survey contained three open-ended questions regarding the 
participants overall satisfaction with MEIMS. All participants made constructive 
criticism, although overall, the response was very positive. The comments indicated that 
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MEIMS is currently a good tool that has the potential to be extremely valuable instrument 
in the prevention of mishaps (See Table 6) 

Question one asked the participant to list the most positive aspects of the 
prototype. Overall, the response was positive. Twelve participants commented to the 
effect that MEIMS would make an immediate, positive impact on safety in the fleet. 

Question two asked for the most negative aspects of the prototype. Most 
comments were problems or suggested improvement to the interface. Such items 
included error checking capability for data input, enlarge text boxes so the data can be 
completely read, line wrap the Contributing Factors text box so that the factors are not cut 
off after one line, and improving the tab function in the main menu, improve the page to 
page navigation. Numerous suggestions were made for “normalizing” the graphing 
functions. Several participants suggested a “find” mode in reports to prevent paging 
through them to find the data desired. Responses produced no discemable difference 
between the tutorial and non-tutorial groups regarding their impression of MEIMS. 

Familiarity and knowledge of HFACS-ME terminology, however, was a distinct 
problem for several of the non-tutorial group. Seven participants made some comment 
regarding HFACS-ME terminology. Most found it hard to understand. Conversely, none 
of the tutorial group, to whom the HFACS-ME terminology was thoroughly explained, 
had a negative comment regarding HFACS-ME terminology. 

Question three requested suggestions for changes to MEIMS. Most participants 
re-iterated their previously mentioned “negatives.” One participant made the suggestion 
that the database should cross-reference the MIR messages for each mishap. Seventeen 
made suggestions regarding navigation, error checking and GUI interface. One 
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participant mentioned making MEIMS available on the World Wide Web. There were no 
noteworthy differences between the tutorial and non-tutorial groups in response to this 
question. 



Table 6: Responses to Open Ended Questions 





Number of Responses 


Theme 


T utorial 


Non-tutorial 


Positive Impact 


5 


7 


Normalize Graphs 


4 


2 


Find Mode in Reports 


2 


3 


HFACS Terminology Difficult 


0 


7 


Improve Error Checking For Data Entry 


9 


6 


Improve Navigation 


4 


6 


Link to MIR message 


1 


0 


GUI Improvements 


8 


9 



E. USER SATISFACTION WITH THE TUTORIAL 
1. Impressions of the Tutorial 

Part IV of the exit survey requested the participant’s impressions of the tutorial. 
Participants responded to five statements using Likert-type responses selecting from one 
of five responses: strongly agree, agree, neutral, disagree, and strongly disagree. Values 
of five through one respectively were assigned to the statements. Additionally, 
participants could make subjective comments on any of the statements (See Table 7) 
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Table 7: Summary of Responses to Tutorial 



Tutorial 


Strongly Agree 


Agree 


Neutral 


Disagree 


Strongly Disagree 


Beneficial 


3 


6 


1 


0 


0 


Made Eval Easier 


6 


3 


0 


, 


0 


Interesting 


5 


5 


0 


0 


0 


Relevant 


j 

8 


2 


0 


0 


0 


Good Concept 


6 


4 


0 


0 


0 



Statement one asked whether the tutorial was beneficial. The histogram of the 
frequency distribution for statement one is presented in Figure 13. The mean for the 
tutorial group was 4.2 (SD = 0.63; RNG = 3). All but one participant in the tutorial group 
agreed that the tutorial was beneficial (n=9; 90%). The participant that was neutral 
commented that a user could figure MEIMS out in “15-30 minutes” without the benefit of 
the tutorial. 




Strongly Agree Neutral Disagree Strongly 
Agree Disagree 



Figure 13: Tutorial is Beneficial 



Statement two asked if the tutorial made the usability test easier to complete. The 
histogram of the frequency distribution for statement two is presented in Figure 14. The 
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mean for the tutorial group was 4.54 (SD = 0.93; RNG = 4). Almost all of the 
participants in the tutorial group (n=9; 90%) considered the tutorial to be beneficial. The 
participant that disagreed restated the reason given in the item above. 
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Figure 14: Tutorial Made Evaluation Easier 

Statement three asked if the participants felt the tutorial was interesting. The 
histogram of the frequency distribution for statement three is presented in Figure 15. The 
mean for the tutorial group was 4.5 (SD = 0.53; RNG = 2). All participants in the tutorial 
group found it to be interesting (n=10; 100%). 
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Figure 15: Tutorial is Interesting 

Statement four asked about the relevance of tutorial to MEIMS. The histogram of 
the frequency distribution for statement four is presented in Figure 16. The mean for the 
tutorial group was 4.8 (SD = 0.42; RNG = 2). All participants agreed that the tutorial 
was relevant to MEIMS (n=10; 100%). 




Strongly Agree Neutral Disagree Strongly 
Agree Disagree 



Figure 16: Tutorial Relevant to MEIMS 
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Statement five asked whether the concept of a tutorial was a good one. The 
histogram of the frequency distribution for statement five is presented in Figure 17. . 
The mean for the tutorial group was 4.6 (SD = 0.52; RNG = 2). Again, all participants 
agreed that the tutorial concept was a good one (n=10; 100%). 
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Figure 17: Tutorial Concept is Good 

2. Open-ended Questions 

The last portion of the exit survey contained three open-ended questions regarding 
the participants overall satisfaction with the tutorial. The participants were asked about 
the positive and negative aspects of the tutorial. Lastly, the participants were encouraged 
to suggest changes to the tutorial. The comments made were overall positive. Some 
constructive criticism was made, aimed at making the tutorial better. 

Question one asked the participant to list the most positive aspects of the tutorial. 
Comments referred to the tutorial as “comprehensive,” “focused on the user,” and 
“providing the necessary background” to take the test. One participant stated that the 
tutorial was thorough without running too long. 
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Question two asked for the most negative aspects of the tutorial. There were 
fewer negative comments than positive ones. They included deleting the historical 
references in the beginning of the tutorial, making the slides "less busy,” and too much 
redundancy of the HFACS-ME codes. 

Question three requested suggestions for changes to the tutorial. There was only 
one comment made regarding changes to the tutorial (n=l; 10%). It was suggested that 
the tutorial should be self-study, providing the participant either the slides (with notes), or 
on a desktop from which to view the tutorial. 
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v. SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 



A. SUMMARY 

Naval Aviation is in the unenviable position of increased demand for high levels 
of operational readiness, equipment availability, and personnel training within a climate 
of reduced budgets. The notion of doing more with less is an everyday reality. To meet 
this demand, asset preservation is vitally important, as funding for replacements does not 
appear likely in the near future. In this fiscally strained environment, the costs of Naval 
Aviation mishaps, in terms of mission capability, operational readiness, and material 
status, are too high. Reducing the number and rate of mishaps occurrence is paramount 
to the success of Naval Aviation. Furthermore, reducing “avoidable” mishaps, those 
involving human error, is an absolute must. Human error in aviation mishaps must be 
identified, and appropriate intervention strategies developed, aimed at reducing the 
causes of those errors. Conventional accident causation theories suggest that accidents 
are the result of a complex combination of errors. Naval Aviation must target 
intervention strategies in key areas to effectively combat human error. 

Initial efforts were made solely in the area of aircrew error reduction, and a robust 
classification taxonomy. Human Factors Analysis and Classification System (HFACS) 
was developed. It was soon realized that to meet the goals in mishap reduction. Naval 
Aviation had to look beyond aircrew error and study maintenance human error. To that 
end a Maintenance Extension, HFACS-ME was built. This expanded taxonomy proved 
to be an acceptable method for classifying human factors in maintenance mishaps. Using 
HFACS-ME to study mishap causes, a clear picture of the human factors contributing to 
mishaps can be drawn. 



To folly utilize the HFACS-ME taxonomy, a prototype database tool. 
Maintenance Error Information Management System (MEIMS) was created. It enables 
better access to the data, query', graphing and reporting functionality, and the ability to 
add new information to the database. In an initial usability study, MEIMS was 
determined to be a valuable tool in the effort to reduce human error in maintenance. A 
MEIMS revision was completed, updating and improving it, and targeting many of the 
issues detailed in the initial usability test. Additionally, a users tutorial was built to assist 
the targeted user in the introduction to MEIMS. This study investigated the validity of 
the MEIMS tool and the tutorial. MEIMS is developing into a robust tool not only for the 
analysis of mishap causal factors, but also an effective weapon in the development of 
prevention programs tailored to human error in maintenance mishaps. Its positive impact 
on the fleet will be immediate and far-reaching. 



B. CONCLUSIONS 

The participants of the study were overwhelmingly supportive of MEIMS and the 
tutorial. Results from the usability test and the Likert-type responses indicate that the 
tutorial was beneficial and made the test easier. Participants in the tutorial group 
performed better in the tasks portion than the non-tutorial group. Based upon the 
independent t-tests conducted, there was a significant performance difference between 
the two groups, with the tutorial group performing better. The response to the Likert-type 
questions did not indicate a significant difference between the tutorial and non-tutorial 
groups (as determined by independent t-test) regarding MEIMS being logical, easy to 
navigate, relevant to maintenance operations and a good concept. Regarding MEIMS 
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being interesting, an independent t-test indicated a significant difference between the 
tutorial and non-tutorial groups, with the tutorial group finding MEIMS more interesting. 
This data indicates an increasing interest level with increased exposure and knowledge of 
MEIMS. The tutorial group’s support for MIEMS was more enthusiastic. Although both 
groups strongly advocated MEIMS relevance to maintenance operations and endorsed it 
as a good concept, the tutorial group was more positive in supporting it. The participants 
of the tutorial provided excellent feedback and also solidly endorsed it as a 
complimentary link with MEIMS. 

All participants made valuable comments and suggestions. Numerous previously 
unrecognized GUI deficiencies were identified. Additionally, data error checking, 
graphing and reporting anomalies were detailed. Most beneficial, however, were the 
suggestions to make MEIMS a better product for the fleet. Specifically linking date- 
time-groups of mishap investigation reports to MEIMS was an especially astute 
suggestion. Also, suggesting MEIMS for the World Wide Web, a pre-planned 
progression for future iterations, was certainly perceptive. 

The potential of MEIMS is expansive. MEIMS should be expanded to include 
other services, allies and NATO mishap information to broaden the scope from which 
data is drawn. Efforts are ongoing to make a variation of MEIMS that would include 
civilian and commercial aviation. That would enable military users an even larger pool 
of information from which to draw data and will be helpful in providing insights and 
parallels between military and civilian/commercial aviation safety. MEIMS should soon 
be available on the World Wide Web. It will be an excellent tool for users, from Naval 
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Aviation’s highest executive levels down to squadron work centers, to derive information 
and to provide the necessary tools to prevent maintenance related mishaps. 



C. RECOMMENDATIONS 

For any software tool to be effective, it must be considered easy to use and 
effective. That notion is especially critical in a tool which the fleet is expected to use. If 
it is not easy to use and effective, it will remain on the shelf. To that end, the following 
recommendations are made for the MEIMS tool: 

• Include the tutorial as a part of MEIMS, upgrading as necessary 

• Provide better navigation from MEIMS functions to the main menu 

• Improve GUI anomalies including the tab function on the main menu, 
improving text boxes so that all data can be read by the user 

• Improve the query causal factors to add text wrap, so mishap information 
will not be cut off. 

• Provide better error-checking during data input 

• Add a find mode in the reports function to enable the user to quickly 
access the desired data without paging through the report. 

• Normalize graph presentations to provide a “weighted” view of data 

• Provide more sub-menus to expedite multi-layer functionality 

• Increase the volume of the database to include more mishaps 

• Provide a link to the date-ti me -group of the Mishap Investigation Report 

• Make MEIMS and the tutorial available over the World Wide Web 
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MEIMS is already a robust tool for analyzing and reporting data regarding 
maintenance human error. Making the above improvements will make it an even better 
tool. Once it is available on-line with the proper training, it should be a well-used tool in 
the effort to preserve lives, material and readiness. 
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APPENDIX A 



T-TEST RESULTS FOR MEIMS EVALUATION AND PART n OF EXIT 

SURVEY 



1. MEIMS TASK PERFORMANCE 

The following is the complete t-test results from the MEIMS evaluation assuming 
equal n and variance. 

Table Al: Test Task Performance Two-Sample t-Test 





Tutorial 


Non-tutorial 


1 

n 


10 


10 


| Mean 


99.10 


93.70 


Variance 


8.10 


36.90 


Pooled Variance 


22.50 




df 


18 




tCrit 


2.10 




tStat 


2.55 




P two-tail 


0.02 





2. LIKERT-TYPE QUESTION RESPONSE 

The following are the complete t-test results from each of the Likert-type 
questions from Part II of the Exit Survey. The t-tests assume equal n and variance. 



Table A2: MEIMS is In a Logical Form Two-Sample t-Test 





Tutorial 


Non-tutorial 


n 


10 


10 


Mean 


4.40 


4.10 


Variance 


0.27 


0.32 


Pooled Variance 


0.29 




df 


18 




t Crit 


2.10 




tStat 


1.24 




P two-tail 


0.23 
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Table A3: MEIMS is Easy to Navigate Two-Sample t-Test 





Tutorial 


Non-Tutorial 


n 


10 


j 

10 


Mean 


4.00 


3.50 


Variance 


0.22 


j 

0.94 


Pooled Variance 


0.58 




df 


18 


t Crit 


2.10 


t Stat 


1.46 


P two-tai! 


0.16 



Table A4: MEIMS is Very Interesting Two-Sample t-Test 
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Table A 5: MEIMS is Relevant to Maintenance Operations Two-Sample 

t-Test 





Tutorial Non-tutorial 


n 


10 


10 


Mean 


4.60 


4.50 


Variance 


0.27 


0.28 


Pooled Variance 


0.27 




df 


18 


t Crit 


2.10 


t Stat 


0.43 


P two-tail 


0.67 



Table A 6: MEIMS Concept is Good Two-Sample t-Test 





Tutorial 


Non-tutorial 


n 


10 


10 


Mean 


4.90 


4.80 


Variance 


0.10 


0.18 


Pooled Variance 


0.14 




df 


j 

18 


t Crit 


2.10 


t Stat 


0.60 


P two-tail 


0.56 
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APPENDIX B 



MEIMS TUTORIAL PRESENTATION WITH NOTES 

Slide 1 



Maintenance Error information 
Management System (MEIMS) 




Human Factors Accident 
Classification System- 
Maintenance Extension 



LCDK Mike •'‘Cracker’* McCracken 
NPS ITM Curriculum 
F-14 RIO 



(Ensure that the students have copies of these presentation slides to use as a 
reference during the presentation) 

Welcome to the Maintnenancee Error Information Management System (MEIMS) 
brief. This prototype tool is built upon the Navy’s HFACS-ME (Human Factors 
Accident Classification System- Maintenance Extension) taxonomy. This 
presentation will detail the HFACS-ME model in detail, explain terminology and 
provide details. Please pay close attention and ask questions as necessary. After 
explaining the HFACS-ME taxonomy, the MEIMS tool, built upon that taxonomy, 
will be briefed and demonstrated. 
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Slide 2 



Naval Aviation Flight Mishap Rate 




Fiscal Year 



FY50-99 



gled Carrier Decks 
Aviation Safety Center 

Naval Aviation Maintenance Program 
FRS Concept Initiated 
NATOP9 Program ' ' 

Squadron Safety Program 

System Safety 
Designated Aircraft 



Naval aviation has become considerably safer over the last few decades, but there is 
still a need for improvement. 

This chart shows the significant reductions in Naval Aviation mishap rates (per 100.000 
flight hours) since 1950. 

Improvements in Naval Aviation safety have primarily focused on engineering and 
administrative controls. They have included (but have not been limited to) design 
improvements (angled carrier decks), the creation of the Naval Safety Center, improved 
maintenance programs (NAMP), training (FRS, NATOPS), local safety programs, and 
system safety designated aircraft. 

The Mishap rate has actually been cut in half each decade since 1 950, HOWEVER, there 
are still many Class A mishaps each year (155 total Class A's for FY 95-99) which result 
in unacceptable loss of life, disabling injuries, and hundreds of millions of dollars in 
damages. So. again, improvements are necessary. 



FY 95-99 


Total Navy/Marine Aviation Class A Mishs 
Flight Hours Flight Mishaps Rate 


FY95 


1 ,656,450 


34 


2.05 


FY96 


1 ,650,026 


36 


2.18 


FY97 


1,523,507 


27 


1.77 


FY98 


1,518,109 


36 


2.37 


FY99 


1,527,186 


22 


1.44 


FY95-99 


7,875,278 


155 


1.97 
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Slide 3 



Engineering & Administrative Controls 
have Impacted Hardware Reliability, but.... 




o - 



^ 



Engineering and administrative controls have significantly impacted hardware reliability (the 
aircraft/machinery are more reliable), but mishaps still occur. Why? 

This chan shows that over the last three decades, the material factors that cause mishaps (e.g., machine 
reliability) have steadily decreased. Human error has also decreased (with training, proper supervision, 
etc.), but at a slower rate so that it has resulted in a higher percentage of the factors that cause mishaps. In 

fact, the percentage of human errors in mishaps is now increasing. 

In 1999. there wore 22 Class A Flight Mishaps; 17 had human error involvement (77%). 

Again, there were 1 55 Navy/Marine Class A’s for FYs 95-99! 
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Slide 4 



Mishap Data Analysis Focus: 

Naval Aviation Class A FM Causal Factors 



70% j 

60% | 

^ 50% i 

i 40% - 
o 

o 30% - 
20 % - 
10 % - 





39% 




initial emphasis on “Pilot Error; 
Hutto reach goal of cutting human 
factors mishaps in half the focus 
has turned & Maintenance 





Determined Causa / Factor 



(NOTE: Mishaps are caused bv more than one factor, therefore, the chart percentages total more 
than 100%) 



To further reduce the mishap rate, we must identify and correct the causes of mishaps. 

Human error is involved in the majority of the determined causal factors in this chart, with material failures 
listed in 39 % of the mishaps. Human error preventative measures have primarily been concentrated on 
aircrew and supervisory issues (e.g., training, SOPs). However, to reach the Human Factors Quality 
Management Board’s goal of reducing mishaps by 50%, we must also focus on maintenance errors. 
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Slide 5 




The Heinrich Ratio 






.5S, 



Fatal Mishap 1 



Non-fatal Mishaps 10 



Reportable Incidents 30 



Unsafe Acts 600 






. 



We have just reviewed some statistics on major mishaps; however, MAJOR ACCIDENTS ARE 
ONLY THE TIP OF THE ICEBURG. 

To make significant improvements in aviation maintenance error, we must thoroughly analyze the major 
mishaps. Fortunately, major accidents (Class A mishaps) are relatively rare events. However, this low 
mishap rate poses a new problem. There is simply not enough information available from Class A mishaps 
to conduct an effective trend analysis. 

Solution: For every' Class A mishap, there are numerous Class B, Class C, and hazard reports which have 
similar cause factors. We must, therefore, thoroughly investigate the “minor accidents*’ as well as the 
“majors” to discover these causes. 

Heinrich Ratio 

The relatively few number of catastrophic accidents are the “tip of the iceberg”. For every major mishap, 
there are 10 less serious accidents, 30 incidents, and 600 hazardous acts. The circumstances (cause factors) 
which raise the severity of the accident are identifiable in all levels of accidents through adequate 
investigation. 

Source: 

Heinrich, H.W., Petersen. D. & Roos. N. (1980). Industrial Accident Prevention: A Safety Management 
Approach (5th ed.) New York: McGraw-Hill. Retrieved from http://hfskyway.faa.gov/training_toc.htm 
(MRM training site). 
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The Costs of Navy 



Maintenance Error Mishaps 



(cost per year in 1997 dollars) 


Class A Mishaps 


$85.0 Million 


Class B Mishaps 


$3.5 Million 


Class C Mishaps 


$3.8 Million 



The Costs of Naw Maintenance Error Mishaps . 

The costs are calculated from the FY90-97 Maintenance Related Mishaps and adjusted to 1997 
dollars. 

NOTE: This only includes component replacement costs and man hours to restore the aircraft in operatin 
condition. It does not include personal injury, loss of life, property damages or liability. 

Class A S85.0 million/vear 

Class B $3.5 million/year 

Class C $3.8 million/year 

Source: Aircraft Maintenance Human Factors PAT report to the Safety QMB. 27 June 1998. 
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Human Factors 
Quality Management Board 




Watershed Event 



Mishap Data 
|1 Analysis 



Recommendations 
for Intervention 



Brainstorming 

Command 



Best Practices 
Benchmarking] 



OK, so we must investigate all mishaps. Are there better or different methods of investigation? Has 
anything been overlooked? 

Occasionally, an accident is so unique and overwhelming that it sends shock waves throughout the 
industry'. These ‘'watershed events” provide a change in focus and are major catalysts for change. One of 
the more famous '‘watershed events” was the 1996 Nashville, Tennessee F-14 mishap. This mishap had no 
material factors... it was a human error mishap. 

Following this mishap, the Commander, Naval Air Forces Pacific (COMNAVAIRPAC) empanelled a 
Human Factors Quality Management Board (HFQMB) to identify human factors threats and develop 
intervention methods to combat them. 

The HFQMB determined that Human Factors was the key to reducing the mishap rate to 1.0 (the 
desired 50% reduction per decade with current rates stagnate near 2.0) 

The HFQMB adopted a three-prong approach to study human errors in mishaps: 

Mishap data analysis to identify hazards and risks 

Benchmarking (of other military' services, NATO partners, private industry', etc.) to uncover 
best practices and find process improvements 
Developing a Command Safety Assessment 

The QMB’s initial efforts were primarily directed at aircrew errors, but the QMB later directed a 
Human Factors in Maintenance and Material (HFAMM) Process Action Team (PAT) to address 
maintenance errors, as well. 
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HFACS- Maintenance Extension 




Management Conditions 





Working Conditions 





Maintainer Conditions 




HFACS was developed to identify factors that relate to aircrew error. The Maintenance Extension 
(HFACS-ME) was added to analyze human error in maintenance related mishaps. So how does 
HFACS identify the factors specifically related to aviation maintenance accidents? 

The "Maintenance Extension"' taxonomy for HFACS was developed to farther classify causal factors that 
contribute to maintenance related mishaps. This addition to HFACS consists of four broad human error 
categories: 

Management Conditions (latent) 

Working Conditions (latent) 

Maintainer Conditions (latent) 

Maintainer Acts (active) 

But does it work ? YES ! HFACS has been applied effectively to the analysis of previous Navy mishaps. 
Additional causal factors were identified, and more importantly, the classification or coding of these factors 
was incorporated into an already extensive database of accidents and incidents. Standardized coding allows 
identification of trends throughout the various accidents and will enable analysts to offer further prevention 
strategies. 

Most important: This POES NOT require a reinvestigation of the mishap . It can be accomplished, 
with minimal training, by investigators or analysts simply coding the narratives and causal factors that 
previously were reported. 

Due to its success: 

- HFACS has been adopted for inclusion into the Naval Aviation Safety Program ( OPNAV 3750.6R) 

- It has influenced policy changes to the Navy's Maintenance Program (4790 series) 

- And is being used to further the effectiveness of Naval Aviation's methods of accident investigation, 
reporting, and database entry. 
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HFACS-ME Framework 







..... . . .... 




Error Categories of HFACS Framework 


First Order 


Second Order 


Third Order 


Management 

Condition: 


Organizational 


> Inadequate Processes - Inadequate Documentation * Inadequate Design 

• Inadequate Rrsova res 




Supervisory 


- Inadequate Supervision 
* Supervisory Misconduct 


- Inappropriate Operations - Cncorrected Problem 


C unCxtxvzs 


Medical 


. Mental State 


• Physical State - Li notation 




Crev' Coordination 


. Communication 


- Assertive mess - Adap tat ilirj/He xftilitr 




Readiness 


- Traizing/Treparaaon, 


- CertijGcation^'Qnalification - Infringement 


Working 

Conditions 


Environ, Tit 


. Lighrhtg/tight 






Equipment 


• Damaged/Ifrtsemced 


- Unavailakkvlnappropriaae - Daub- Decertified 


. ' : ■ ; 


Workspace 

. 


- Cor, firing 


• Obstructed - laarcesstble 


UaintxxaerActs 


Error 


- Ataratran/Meawry 

- SldOCTeclunque 


- JudgaentDeasam .\lakzsg - Knowledge Rule Based 




Violation 


UKouxtne 
| flagrant 


~ Infraction -Exceptional 








i 



The HFACS-ME taxonomy also includes three orders of error: first, second, and third that reflect a 
macro to micro perspective. 

The levels provide sufficient description to develop a searchable database and are useful for distinguishing 
supervi sory areas of action. For example, a First Order category could be briefed at the Executive level a 
Second Order at the Vice President/Senior Manager levels, and the Third Order can be corrected by Safety 
Officers and front line supervisors. 

An Example: Maintainer Conditions (First Order), Readiness (Second Order), and 
Qualification/Certification (Third Order) 
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HFACS - Maintenance Extension Model 




Supervisory Mnititnlnnr Wor?cist^ 

Conditions Conditioort H| Conditions 




This model provides a realistic example of the interaction between the Error Categories. 

Supervisory Conditions, Maintainer Conditions, and Working Conditions may 
INDEPENDENTLY or COLLECTIVELY affect the Maintained Actions. 

The Unsafe Maintainer Act which follows may either: 

(1) Lead directly to a mishap or injury (Example: a maintainer runs a forklift into 
the side of an aircraft and damages it), or 

(2) Become a latent Maintenance Condition, which the aircrew would have to 
deal with on take-off, in-flight, or on landing. (Example: an improperly rigged landing 
gear that collapses on touchdown or an over-torqued hydraulic line that fails in flight and 
causes a fire) 

It is important to note that Supervisory' Conditions related to design for maintainability, 
prescribed maintenance procedures, and standard maintenance operations could be 
inadequate and also lead direcdv to a Maintenance Condition. 

Please note that this model utilizes only the First Order Error Categories (again, 
shown in red). So how does the Second Order Error Categories affect this 
relationship? (quickly go to next slide) 
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HFACS-ME Framework 


1 ' 














Error Categories of HFACS Framework 


First Order 


Second Order 




Third Order 




Supervisory 


Organizational 


- Hazardous Operations 


- Inadequate Design 


- ^adequate 


Condsions 




Documentation 










- Inadequate Resources 


- inadequate Processes 








Squadron 


- Inadequate Supervision 


* Inappropriate Operations 


- Unconeded Problem 








- Supervisory Misconduct 






Working 


Environment 


- Ugtnmg/Light 


- Weatner/Exposure 


• Environment?" 


Conditions 
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Equipment 


- Damaged 


• Unavailable 


- Oated/U needled 
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* Obstructed 
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Error 
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The Second Order of Error Categories (shown in Blue) is simply a further breakdown, or description, of the 
First Order Error Category (so additional relationship models are not necessary). If we look back a couple 
of slides... (quickly go to next slide)... 
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HFACS- Maintenance Extension 



5 



Supervisory Conditions 



Orsatiizadotiai Squadron j 



Working Conditions 
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■ — ■ 





Maintainer Conditions 




^ Medical \ Crev* Coordination jj 


leaduicss f| 






’ r, .. ; 1 

Maintainer Acts 




Violation 






ISBAP/INJ8R) 
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...the incorporation of the Second Order Error Categories would look like this. 

Supervisory Conditions are established at the Organizational levels (outside your command) or at your 
Squadron level. 

Working Conditions that affect one or more maintained include the Environment. Equipment, and 
Workspace limitations. 

Maintainer Conditions directly impact an individual maintainer and may include Medical, Crew 
Coordination, or Readiness factors. 

Maintainer Acts can either be Errors or Violations. 

(NOTE : Multiple Error Categories may be factors in mishaps. For example, both Organizational 
and Squadron practices may combine to form Supervisory Conditions that lead to mishaps, or a 
combination of Equipment and Workspace conditions may promote an unsafe Working Condition.) 

Now, as you probably surmised, the Third Order Error Categories are simply a further breakdown 
of the Second Order Error Categories shown on this slide. 
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Here again, are the Third Order Error Categories (now displayed in Green). 

The most effective way to define and display the relationship of First. Second, and Third error 
categories is to examine each First Order Category independently, (quickly go to next slide) 
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HFACS- Maintenance Extension 
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We will begin by examining the Supervisory Conditions First Order, (quickly go to next slide) 
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Third 

Orders 



[Hazardous Operations! 








Failed 




Poor Risk Management 
Operational Doctrine Errors 
Failed to Provide Training 


imMsmsarn 






I Inadequate Documentation 


j Inadequate Resources! 



j Inadequate Design" 



Purchasing Failure 
Deficiency not Corrected 
Modified Equipment 



[Inadequate Processes! 



No/Poor Documentation 
Documents not Updated 
Documents unusable (CDs) 



Improper Manning 
Lack of Funding 
Lack of Pans/Equipment 



Fail to Enforce Regulations 
Failure to Provide Oversight 
Failed to Track Performance 



Under Supervisory Conditions, we have both Organizational and Squadron Second 
Order Error Categories. 

We will start with the Organizational Category. 

(Facilitator note: Read each term and its example below and then mention some of 
the additional examples listed in each box on the slide. Encourage discussion from 
students) 

Under the Organizational Second Order, we have the following Third Order Error 
Categories : 

Hazardous Operations An engine that falls off of a stand during a change out evolution 
due to an unforeseen hazard of a high seas state 

Inadequate Design The poor layout of system components that do not permit direct 
observation of maintenance being performed 

Inadequate Documentation A manual omits a step in a maintenance procedure, such 
as leaving out an o-ring that causes a fuel leak 

Inadequate Resources Insufficient funding, manpower, tools, parts and equipment to 
perform maintenance effectively and safely 

Inadequate Processes Limitations in teamwork, communication, and directives within 
and between organizations 

(NOTE: The descriptions given, and the examples shown on the slide, are just that . 

. . examples. The purpose of this presentation is to establish a knowledge and 
understanding of the HFACS-ME Categories, not restrict their use to only a handful 
of scenarios .) 
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Fail to Provide Guidance 
Fail to Provide Oversight 
Fail to Provide Training 
Fail to Track Performance 
Fail to Track Qualifications 



Failure to Inspect 





Insufficient Resources Documents not updated 

Inadequate Brief Times No Corrective Actions 

Improper Manning Unsafe Condition not Reported 
Inadequate Schedule 



Ignoring Risks 
Fail to Enforce Rules/SOPs 
Use of Unsafe Equipment 
Use of Untrained Personnel 



And under the Squadron Second Order Supervisory Condition, we have these Third 
Order Error Categories: 

Inadequate Supervision A supervisor who does not ensure that maintenance personnel 
are wearing required personal protective gear 

Inappropriate Operations A supervisor who directs a maintainer to perform a task 
without considering risks, such as driving a truck through a hangar 
Uncorrected Problem A supervisor who neglects to correct maintained who routinely 
bend the rules when they perform a common task 

Supervisory Misconduct A supervisor who willfully orders a maintainer to wash an 
aircraft without proper safety gear 
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HFACS- Maintenance Extension 



Supervisory Conditions 
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Let us now examine the Working Conditions First Order Error Category breakdown. 

(Again advise students to follow along with their copy of the HFACS-ME Framew’ork slide), 
(quickly go to next slide) 
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Working Conditions ■ Third 

IHHHfe I Orders 



f Lishtins/Lieht 


i Weather/Exposure ! 


(Environmental Hazards! 


Night visibility 
Workspace Illumination 
Inadequate flashlights 


Extreme Temperatures 
Ice on Equipment 
Visibility in Rain/S now/Fog 
Equipment/Manning Changes 
Inadequate Clothing 


Slips/Falls 

Frostbite 

Sunstroke 




Working Conditions have Second Order Error Categories of Environment, 
Equipment, and Workspace factors. 

The Environment Second Order can be further broken down into the following 
Third Orders: 

Lighting/Light A maintainer who is working at night on the flight line does not see a 
tool he left behind 

Weather/Exposure A maintainer who is securing an aircraft in a driving rain fails to 
properly attach the chains 

Environmental Hazards A maintainer who is working on a pitching deck falls from the 
aircraft. 

(Note: Environmental Factors are often not adequately reported.) 
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Working Conditio n s ■ Third 

HBl BPr lOrders 



Damaged 


1 Unavailable j 


[ Datcd/Unccrtified | 


Limited Use 
Unreliable 

Gauge/Calibration Errors 
Unsafe (brakes, electrical) 


Equipment used elsewhere 
Equipment not in inventory 
Equipment unusable 


Calibration Expired 
Open Purchase/Uncertified 
Extended Beyond Service Life 




Equipment Third Orders are: 

Damaged A maintainer who is using a defective test set does not pre-check it before 
troubleshooting 

Unavailable A maintainer who starts working on landing gear without a jack because all 
are being used 

Dated/Uncertified A maintainer who uses an old manual because a CD-ROM reader is 
not available. 

(Note: Tools and Equipment Factors are also not adequately reported.) 
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Working Conditions 






J 



Third 

Orders 



Confining 


1 Obstructed 


Inaccessible 


Insufficient Workspace 
Insufficient Maneuverability 


Vision Obscured (fog,$moke) 
Vision Blocked (obstacles) 
Maintenance Hindered 


Inadequate Aircraft Design 
Inadequate Support Equipment 






And finally. Workspace factors can be separated into these Third Order Error 
Categories: 

Confining A maintainer who is working in a hangar bay cannot properly position the 
maintenance stand 

Obstructed A maintainer who is spotting an aircraft with his view obscured by catapult 
steam 

Inaccessible A maintainer w'ho is unable to perform a corrosion inspection that is 
beyond his reach. 

(Note: Workspace factors are also not adequately reported.) 
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Maintainer Conditions, the conditions that uniquely affect individual maintainers, are separated into 
Medical, Crew Coordination and Readiness Categories, (quickly go to next slide) 
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Third 

Orders 



Mental State 


1 Pbvsical State 1 


Complacency 
Distracted 
Mental Fatigue 
Life Stress 

Misplaced Motivation 
Task Saturation 
Loss of Situational Awareness 
Channelized Attention 


Medical Illness 
Physical Fatigue 



j Limitation 



Insufficient Reaction Time 
Visual Limitation 
Incompatible Intelligence 
Incompatible Aptitude 
Physical Capability 
Physical Reach/Size 



The Medical Third Orders are based on an individuals mental and physical abilities 
(and limitations) to perform a task at a certain time. 

They include: 

Mental State A maintainer who has a marital problem and cannot focus on a 
maintenance action 

Physical State A maintainer who worked for 20 hours straight and suffers from fatigue 
Limitation (Physical) A maintainer who is short cannot visually inspect an aircraft 
before it is launched 

(Note: Medical Factors are not reported adequately) 
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Third 

Orders 



{ Communication j 

Terms not Standardized 
Hand Signals not Standardized 
Documentation Failures 
Documentation Delays 
Equipment Failures (radios) 
Equipment use (lights/whistles 
Inadequate Brief 



\ Assertiveness 

Fail to Brief/Make Suggestions 
Fail to Correct Discrepancies 
Fail to Confirm Messages 
Inattention to Feedback 
Waiver when Confronted 





Emergency Response 
System Failure Response 
Changes to Routine 
Team member changes 



mm® \ 




Crew Coordination factors include problems in: 

Communication A maintainer who leads a taxiing aircraft into another due to improper 
hand signals. 

Assertiveness A maintainer who performs a task, not in accordance with standard 
procedures, because the maintainer was overly submissive to a superior. 
Adaptability/Flexibility A maintainer who downplays a downing discrepancy to meet 
the flight schedule. 
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Infringement 



Not Trained for Task 
Not Current in Training 
Unrealistic Training 
Ineffective OJT 



Not Certified in Task 
Not Certified in Model 
Qualifications Expired 



Intoxicated at Work 
Hung over 
Drug use 




And the last Maintainer Conditions category to discuss is Readiness- Its Third 
Orders involve: 

Training/Preparation A maintainer who is working on an aircraft skipped the requisite 
OJT evolution 

Certification/Qualification A maintainer who engages in a procedure that he or she has 
not been qualified to perform 

Infringement A maintainer who is intoxicated on the job. 

(Note: Readiness Factors are also not reported adequately.) 
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HFACS- Maintenance Extension 






Maintainer Acts 



The final category is Maintainer Acts, which include Errors (mistakes) and Violations (willful acts), 
(quickly go to next slide) 
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Third 

Orders 



f Attention j 

Missed Communication 
Loss of Situational Awareness 
Distracted 
Fail to Secure 
Left Unattended 
Improper Lookout 



Knowledge/Rule based > 



Insufficiently Trained 
Insufficient References 
Incorrect Procedures 




I Skill Based j 

Delayed Response 
Overuse of Controls 
Omitted Step in Procedure 
Omitted Checklist Item 
Poor Technique 
Improper Cross-Check 



Memory 



Procedure Mistakes 
Missed Task 
Sequence Errors 



Judgment/D ecision-Making) 

Exceeded Ability 
Poor Decision 
Wrong Response 
Misdiagnosed Problem 
Improper Procedure 



Maintainer Errors are of the following Third Order types: 

Attention A maintainer who misses a hand signal and backs a forklift into an aircraft 
Memory A maintainer who is very familiar with a procedure may reverse steps in a 



sequence 

Know ledge/Rule Based A maintainer who inflates an aircraft tire to a pressure required 
by a different aircraft 

Skill Based A maintainer who roughly handles a delicate engine valve will cause undue 
damage 

Judgment/Decision-Making A maintainer who fails to make appropriate decisions due 
to limited information, inadequate preparation, or perceived pressure to perform 
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Maintainer Acts 



Third 

Orders 



Routine 




Infraction 


Did Not Follow Brief 
Bending of Regulations/SOPs 
Use of Incorrect Equipment 
Violated Training Rules 
Do Not Utilize Checklists 


it; - ^ ^ | 


Single Event to Save Time 
Violation to Expedite Mission 
Use of Incorrect Equipment 
Skip Publication* Crosscheck 


1 Flagrant 


^ jt'f; 


t Sabotage 1 


Unauthorized Procedure 
Not Current/Qualified 
Exceed Equipment Limits 
Thrill Seeking 


m 


Deliberate destruction 
Attempt to injure/kill 
Purposeful Mission Disruption 



There are also several types of Maintainer Violations. Their Third Order category 
is based upon frequency, intent, and supervisory involvement. 

Routine A maintainer who engages in practices, condoned by management, that bend 
the rules 

Infraction A maintainer who strays from accepted procedures to save time, bending a 
rule 

Flagrant A maintainer who willfully breaks standing rules disregarding the 
consequences 

Sabotage A maintainer who performs a criminal act to destroy equipment or endanger 
the lives of others 
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(FY 90-97) Human Error in 
Maintenance Related Mishaps 



> 470 Maintenance Related 
Mishaps Were Analyzed For 
Human Errors 

> Classification Process 
Performed by Naval 
Maintenance Personnel 




HFACS, as previously stated, has been used effectively within the Navy. Without reopening an 
investigation, HFACS was applied to existing Mishap Investigation Reports. 

(go to next slide for general findings). 
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General Findi ngs 

>Poor/non-existent maintenance 
procedures 

> Inadequate/poor supervision of 
maintenance evolutions 

> Miscommunication - supervisor 
to subordinate, pass-down, or 
shift turnover 

>Not using, lack of, or outdated 
publications 

> Violations - not following policy, 
procedures, checklists, etc. 




This slide shows the general findings of the analysis. Fm sure we are all familiar with these latent 
conditions. Let us now look at another study of 63 Class A mishaps between FYs 90-97. (go to next slide) 
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HFACS-ME Results of MRMs 




HFACS ANALYSIS OF Class a FYs 90-97 Maintenance Related Mishaps (MRMs). 

During FYs 90-97 there was a total of 63 Class A Mishaps, of which 61 were Flight, 0 were Flight Related, and 2 were 
Aircraft Ground. Two Navy Maintenance Officers and two Navy Chiefs used the HFACS Maintenance Extension to 
classify the human factors causes reported in these mishaps. They discovered the following profile of human errors: 
Management Conditions -679c of all Naval Aviation Class A MRMs reported Squadron Supervisory 
Conditions, whereas 21% had Unforeseen Supervisory* Conditions (not shown). 

Maintainer Conditions - 21 % of all Naval Class A MRMs reported Medical, CRM. or Readiness Maintainer 
Conditions. Note: Maintainer Conditions were under reported , more are likely present and have an effect 
Working Conditions - 3% of all Naval Class A MRMs reported Environment. Equipment, or Workspace 
Working Conditions. Note : Workspace Conditions were under reported* more are likely present and have an effect. 
Maintainer Acts - 75% of all Naval Aviation Class A MRMs reported Maintainer Errors, whereas 40% had 
Maintainer Violations. 

Clearly, latent conditions in the form of Supervisory, Maintainer, and Workspace factors are present that can impact 
maintainers in the performance of their jobs. However, many Maintainer and Workspace Conditions arc not reported 
due to the reporting system in place, perceptions of accident causation, or culturc/climate issues. Specifically, 
inadequate supervision of maintenance evolutions, not ensuring personnel are trained and/or qualified, not enforcing 
rules, and poor communication characterize the majority of latent Supervisory’ Conditions. Poor pass down, 
coordination, and communication; non-use or lack of publications, policies, and procedures: and fatigue comprise most 
latent Maintainer Conditions. Finally, most Maintainer Errors reflect a lack of training, experience, and skill, whereas 
Maintainer Violations consist of routine non-compliance with standard procedures and practices, and infractions, 
bending the rules in order to meet mission requirements and the flight schedule. 

Conclusions 

HFACS-ME was effective in capturing the nature of and relationships among latent conditions and 
active failures present in Class A MRMs. The insights gained provide a solid perspective for the 
development of potential intervention strategies. The major mishaps analyzed were primarily FMs, meaning 
that many imposed in-flight Maintenance Conditions on aircrew. During FYs 90-97 there were almost 500 
MRMs in Naval Aviation, many of which were of lesser severity and were either Flight Related or Aircraft 
Ground Mishaps. Such mishaps involve primarily ground and ramp activities and can lead directly to a 
mishap or injury. Consequently, the present profile and observed relationships only hold for the mishaps 
considered and cannot be generalized to all MRMs. Further, it can be contended interventions developed for 
major mishaps that primarily involving maintenance activities such as engine repair are likely not appropriate 
for ones of lesser severity that involve other activities such as loading ordnance or towing aircraft. 

Presently, an in-depth analysis of all MRMs is underway and it is planned to contrast major vs. minor 
MRMs, MRMs occurring during maintenance, stores, and ramp activities, and MRMs for specific 
communities. 
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HFACS-ME Summary 

*/ 



> HFACS-ME is effective in classifying 
aviation maintenance causal factors 

> HFACS-ME enables organizations to 
develop successful accident intervention 
strategies 

> HFACS-ME should be applied to both 
major and minor accidents to fully analyze 
maintenance errors 



Presentation Summary: 

The HFACS-ME model, as you have just seen, has demonstrated effectiveness. 

With HFACS-ME, a safety organization can: 

- effectively classify (code) maintenance related accident causal factors 

- develop specific intervention strategies to mitigate causal factors 

NOTE: Because most major mishaps occur during flight operations, it is essential to also evaluate the 
''minor' mishaps and incidents that occur on the ramp and in the hangar. Such mishaps involve activities 
that can lead directly to damage to the aircraft or injury to the maintained 
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Maintenance Error Information 
Management System (MEIMS) 



- ■ . . jBbWfi a,ia^ jjwaiL ^ « 



■ HFACS-ME 

■ Taxonomy 

■ Database Tool 



■ Prototype developed 

■ at NPS 



US Naval Aviation 
Maintenance Error 
information Managtnenr 
System' (MEIMS)' ^ 4 



3 Cuery Mcjxji 
_j export Ctsp>f> Momt 
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j ACd Nov Data. 
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Second Revision (Nov 2000) 



■ 






Now that we have some understanding of the HFACS-ME, lets take a look a the 
prototype database tool which uses that taxonomy* It is called MEIMS, 
Maintenance Error Information Management System. 

This is an Access 2000 based prototype, developed at NPS, the latest revision just 
recently completed. 
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US Naval Aviation 
Maintenance Error 
Information Managment 
System (MEIMS) 




a 


Query Menu 


J Add New Data 




Expert Graph Menu 


I Investigation Tool 




Report Menu 


_J Exit 



The Main Menu 

Notice the five button functionality, plus exit. Placing your pointer over any 
of the menu selections will give you a brief description of that selection. 
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Queries j 



Select Mishap Data By 



i"l Aircraft Model 

_J Aircraft Type 

» ACAIR. He!o.. 

| Branch of Service 

USN. USMC 

j Location of Mishap 

Ashore. Embarked... 

J Mishap Class 

A3.C 

_J Mishap Type 

FM. FRM. AGM 

! Year of Mishap 

Calendar 



j Multiple Criteria 

Select or more 

criterion trom the left 



J H FACS- ME Summary 

by ME Category using a 
Multiple Catena Query 

j Return to Main Menu 






The query menu. 

The user may select one or many (using the Multiple Criteria button) choices 
from the left and get mishap information regarding that particular selection(s). 
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5E Summary of ffehap 



Maintenance Mishap Query Close Form 



Mishap Number 


IT 


Class of Mshap 


r 




Date or Mishap 


J1/28/799 


Branch of Service 


[USN 




Aircraft Type 


piT - 


Aircraft Category 


cc 

1 




Mishap Type 




Location of Mishap 


(Ashore 


*>- - Vil t ,y>; 



Brief Desertion 



[Aclt on FCF had left eng oi hot ight during dim 



CoobbUbg Factors 



'DefmeriFACS Codes; 



levefl 



Work cfe supv. and C01 fakd fo provide adequate tech data and procedures due to corapiacer SC 



Level 2 Level 3 
SON IDS 



<Back j 

Record: M I j I F T . * J ►! \±*1 of 347 (rfcered) 

This is the TACAIR selection. Notice that the selected criteria is in blue. There is a 
brief description, contributing factors and the codes for Levels 1, 2 & 3. Definitions 
for those codes can be found by clicking on the “Define HFACS Codes” button. 
Also, at the bottom, the number of mishaps meeting that criteria. 
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Select Multiple Criteria Query 



Aircraft Model 

F Aircraft Type jHeio y] 

r Branch of Service 
r Location of Mishap 
F iMishap Class t (a yj 
r Mishap Type 
r Year of Mishap 
< Back | View Selection | 



Click the box 
next to the 
criterion that you 
wish to select 

Select the 
desired item 
from tiie drop 
down box. 



This is the Multiple Criteria Selection screen. Again, the mishaps meeting the 
selection criteria are viewed... 
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53 Summary of Mishap 



Maintenance Mishap Query 



Mishap Number 


F 


Gassof Msfcap 


Date of Mishap 


jv 23 /JS 3 


Branch of Service 


Aircraft Type 


|hT“ 


Aircraft Ceteqcxy 


Mishap Type 


JrM 


Location of Mishap 



End Description 



JDjxj 

Close Fcrai | 

r 

{usiT" 

(h 5 o 

(Ashore 



[WC EXP HI YAV.STRKGRD. IDG COLLAPSED l TAIL HIT 3 RD 



CbntnUhg Facta: 



"D efoe KFACS C odec 



MCO Failed to Provide ST System to Document A/C S Sates in AD 8 
1 NADQ Design of Sonar Seat Saitoh Afcws Salt Water InSridcrXonosion 



LeveH Levd2 Level 3 

SC SGN IDO 

SC GRG DE 3 



<Back | 

Record: JiLlJi r jJ±lkdcf 19 (Stored) !• ... - HI HE* — , 



...with the selection criteria in blue. Define HFACS Codes allows the user to view 
the codes for each level... 
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Li* v r* ~r; ■yr'n 



C't ‘Art 7 c2 -w- • 



*'125* t£3 '-V 



Supervisory 
Conditions (SC) SqM#<»r««(S«0 



Organizational 


Hazardous Inadequate Doc- 


Inadequate 


Inadequate 


hadequate 


(ORC) 


| Operations (HAZ) umer&Son(DOC) 


Design (DES) 


Resources (RES) 


Processes (PRO) 



^adequate 



Inappropriate 



Uhoorreded 



Stjpervfcory 



Supervision (DQ) Operations (OPS) Probiem (PRS) [ tf$conduct(M(S) 









wersal State (fcNT) Physical 

State (PHY) 


limitation (LW) 


fetantiiner 
Conditions (MC) 


Crew 

Coortfirmtion 




Corrwfcadon Assertiveness 
(COM) (ASS) 


Adaptafc&y/ 

Fte?*ility(ADA) 


-jr~' " ' * <CKW> 





I 




m 


Environment 

©TO) 




lighting; Vfeather/ Envionraertal 

Licrt(LGT) Exposure (WXE) Hazards (EH2) 


Ytortdng 
Conrtftkres ft¥C) 


Equipment (EQP) 




Damaged (Dfcte) Uwrvaiabie (UNA) Dated/ 

Uncertiffed(DUC) 


H| 


Workspace 

(VttfQ 




Confining (CON) Obstructed (Q9S) toaccesssbfe(N«,) 



Readiness (ROY) ■ Training/ Certificabonf Infringemert (WF) 
I Preparation (TKG) Qualifcston (CRT) 



n Dose,fprm: 



Error (ERR) ■ «tertion(ATT) Memory(*&0 Knowledge/ sa3ased(SKL) I Judgeswt/ 
I ais-Sased (K?W) | Decisioo-fctefcrig 



traveling right to left in Levels 1, 2 & 3 
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And Selecting HFACS-ME Summary gets the user... 
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EE 






13 vr»» £. fCor< ^ Uro 

H - y cS 3 l v i ^ s? <? v ^'/.i. : v^7 a >* ** 



3 . 




I^M 9 Q|M 0 D vOO* 

UHHrtMian C>CO 




»W*nWl |Mrean*cSOK 5 > 




| UtfatoriGJM) 

s«*?W I 



'***£ tt*r*e 6 **v. A. 3 , C 



1 Cowrxraww* 1 
1 <«K 1 


1 ^sartwwa, 1 
I (ASS> I 


| A*0M*v/ j 
| n^y*y<A©A) 


0 


0%- 


0 


cr* 


0 


0% 


1 liirW 1 
|Prep«sc*»flr*& | 


I Cortteaberw 1 


1 rnnoeew* fWT* 


0 


o% 


0 


0% 


0 


c% 


I Up 

I w* 


^ 1 

O-OTl I 


1 ^ 

I &900M 


ttr/ 1 
r*rwe 1 


1 ^ 

| mart 


wc*W 
t»£EM2> | 


0 


0% 


0 


C% 


0 


C% 


1 ?***?&&*», 1 


1 jn e»!*o 


Be.,Afc, 


1 0-tef 










1 IXeffK^od CDiX) 




11X 


0 


c* 


0 


c*»» 


| COftOnnstCOfC) | 


k«sn<0MW 1 




0 


0% 


9 


Ox 


0 


cx 


BBS! 




I KW 

L 


**>*■'' 




tr*. 


0 


Ox ^ 


2 


22* _ 



A<n*afc W«x*eJ 




.=1 


Ajro«an Type 


P~ 


-U 


$«v»C* 




jJ 


Locjfcon o < M*xh«c 






Mctfcap data 


ST3 


Type 


F“ 


-li 


tVj* oJ Mishap 




-d 



T o> > Nurtjw o' *<F V«hflce 

9 



Ops-eFc**^ 



S»9M(S^)|D!)(t9tM«i^ 

I (JOC, 



,{tf? x: 

■I4125I 



*1 

1 



^Mort y! ,^S OS ^ , ^k>o^ot. ^6o»wx— _ jJ&jeten jgnwiMarn. SjosaaJKv 31 vupvMI,,.. ’^Allv^rtv, 






...This presentation. Notice that the user can modify the choices directly from this 
screen (F-14/Class B selected). This summary page is good for pulling out data 
specific to levels 1-3 or within levels. 
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.in; : 



US Naval Aviation 
Maintenance Error 
Information Managment 
System (MEJMS) 










lf.S* 

: 

£3 Query Menu 
• , 

1 Expert Graph Menu 

I Report Menu 



I Add New Data 

! Investigation Tool 

_J Exit 



Returning to the Main Menu, we will now look at the Graphing Function 
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[si Expert Graph 










Expert 


XAxis 


YAxis 






Graph Menu 


s; : 


C* 


.Aircraft Model 






r 


a 


Aircraft Type 


1 


1 .Select X-Axis and YAcs 




Mishap Location 




categories by checking the 




r 




option button to the left of 
the category 


r j 

1 


r 


Service 




Z Select specific 


r 


r 


Mishap Qass 




values for X and Y-Axes by 










clicking on the "Select Axis 
Values’* command button 


r 


r j 

\ 


Type of Mishap 




at the 




1 






bottom. 

3. Click on the "Graph 


c 

; 


r 


H FACS -ME Level One 


1 


It" command button to 
create the graph (w21 not 
be available until both axis 


1 

r 

r 


r ! 

i 

r 


HFACS-ME Level Two 






H FACS -ME Level Three 


i 

1 




The Graphing Function allows the user to choose which criteria will be on which 
axis, and then... 
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■■■■■I 



Expert 
Gra 



X Axis Y Axis 



@ Multipie Selection 



1. Select 
categorie 
option bu 
the catec 


Available 




r A4 


GJ 


AV8 
Cl 2 


□ 


2 


Cl 30 




values fo 


C2 




cEcking c 


C9 




Values" c 


£2 




at the 


ES 




bottom. 


EAS 




a CE 


FI 4 




It" comm. 


F4 




create th* 


F5 


jii 


be avail a 





J*J 



Selected 



2ll 

J3 



OK 



] 



Cancel | 




S elect X Axis Values 



< Back 



S elect Y Axis Values 



Graph It 






Both the X and Y axis may be further defined to produce.... 
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i H* Vew insert Forr^t R score? ?x$ Wndow He£ 



M' S §Qlv- 



14 ►- g 5 5 '3 * 13 < 



3D 

25 

> 28 
u 

I 15 
5 

£ 18 
5 
8 



<3ad 



...the selected graph, in this case, F-14 and F18 class A and B mishaps. 
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US Naval Aviation 
Maintenance Error 
Information Managment 
System (MEIMS) 






I Query Menu 

' .: r' " ' s’W- - 

I Expert Graph Menu 

O Report Menu 




I Add New Data 

I investigation Too! 

! Exit 



Selecting the REPORT MENU gets... 



107 



Slide 46 






Report Menu 



Reports | 



^ "SSWJi 



. 



Select A Report | 



-J 


Mishap Distribution - All Mishaps 


-1 


Mishap Aircraft Distribution 


-J 


I 

Mishap Class Distribution 


Q 


Mishap Type Distribution ■“ 


_j 


Mishap Ciass/Type Distribution 


m 


Mishap Distribution - Service 


□ 


Mishap Distribution - Location 


_j 


All Mishaps-Chronologica! Listing 

(by Aircraft 1 S83-1 3S9; M aintenance oniy) 


_j 


Return to Main Menu 






...a choice of multiple reports 
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All Maintenance Related Mishaps by Branch of 
Service <1 989-1999) 

3 UNK 

Unsafe Supervisory Conditions (USQ 2 67% 



1 


33% 


Organizational 




t ocrx> 


Hazardous Unsafe Operation 


1 


io o% 


Inadequate Documentation 


o 


0% 


Inadequate Oesiqn 


o 


o% 


Inadequate Processes 


o 


crx> 


inadequate Resources 


2 


67% 


Squad ron 




so-% 


Inadequate Supervision 


o 


o% 


Inappropriate Operations 


1 


50% 


Uncorrected Problem 


Ov 




Supervisory Wisconduct, 


Unsafe Maintainer Conditions (Ut!lC) 


1 33% 


1 


33% 


Medical 


1 


'100% 


Physical State 


O 


o% 


Mental State 


O 


o% 


Umitaiion 


o 


o% 


Crew Coordination 


o 


Cr% 


rtive ness 


o 


o% 


C omm unication 


o 


o% 


/Ada pt a b U tty/F ie x ib i lit y 


o 


o% 


Readiness 


o 


o% 


T raimnc./Preparation 


o 


o% 


C e rtifi ca ti o n/O u a title at io n 


o 




Infrin gement 


Unsafe Working Conditions (UWQ 




o o% 


O 


o% 


Environmental 


o 


o% 


Eauromen: 


o 


o% 


Workspace 


Unsafe MdintainerActs (UMA) 




1 33% 


1 


33% 


E rror 


-l 


lOOf% 


rvt ia n 



...in this case, the Mishap Distribution by Service. 
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US Naval Aviation 
Maintenance Error 
Information Managment 
System (MEEV1S) 



_J 




Expert Graph Menu 
Report Menu 



Investigation Tool 
_] Exit 



The Add New Data button produces... 
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Enter New Maintenance Mishap Date Add New Record [ Cbserorrn 



J 



Mishap ID | SCO 


Mishap Gass) 8 


to 


Mishap Type FM 


to 


Date of MrshapjlZ'25/2000 


Aircraft Type |H-53 


to 


Aircraft Category jHdo 


tol 


Branch of Service jUSN 


to 



Descriptor* 



Mid-air with reindeer drawn slercKj 



to 



Record: i< M 1 T > i M I>-^1 of ! 



Add New Factor 



Mishap ID 
FACTORS 



factors ID 



HFACSCode 

Definitions 



3rd Level Code r~n r 



i.-Xr '#** A i •»««. „ »5- 



to 



3rd Level Description j 

2nd Level Code j~ 
2nd Level Description J 
1st Level Code f 
IstLev^ Description 



Reccr ^ : n 1 1 1 i r » i ►» i i of i 



A form on which to enter mishap data. In this case, it seems that some of Santa’s 
helpers may have run into some trouble. 



Again, on this form, HFACS Code Definitions are available. 
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US Naval Aviation 
Maintenance Error 
Information Managment 
System (MEIMS) 



-1QS X? 






_l 

J 

J 




Query Menu 
Expert Graph Menu 
Report Menu 



_J Add New Data 
{3 Investigation Tool 
_J Exit 



The Investigation Tool is not yet fully functional, but will allow the user to “drill 
down” into the specifics an individual mishap to better ascertain learning point. 

Last, but not least. Exit. This button exits the MEIMS program and Microsoft 
Access 2000. 
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This concludes the Tutorial. You now have a better understanding of the individual 
error categories and their interrelationships in a mishap’s chain of events. 
Additionally, you have seen some of the functionality of the MEIMS tool. If there 
are no questions, please follow the instructor’s directions regarding the usability 
test 
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APPENDIX C 



MAINTENANCE ERROR INFORMATION MANAGEMENT SYSTEM (MEIMS) 

EVALUATION 

Background. Thank you for participating in a usability study (evaluation) of a 
prototype tool for the Maintenance Error Information Management System (MEIMS). 
This tool was developed at NPS and has been modified based upon previous usability 
studies. This study is being conducted by LCDR Mike McCracken, USN as part of a 
thesis project for his Master of Science program in Information Technology 
Management. MEIMS was developed to address and identify patterns of human error in 
Naval Aviation maintenance-related aircraft mishaps. The Human Factors Analysis and 
Classification System Maintenance Extension (HFACS-ME) taxonomy is the foundation 
of MEIMS and is an effective method for classifying and analyzing the presence of 
human error in maintenance operations leading to major mishaps, accidents of lesser 
severity, incidents and maintenance related personal injury cases. A previous study 
indicated that prior to using MEIMS a tutorial, explaining the taxonomy and some 
features, would be beneficial. This study compares two groups of users, one which 
receives the tutorial and one that does not. Feedback from user groups will be used to 
further improve ME IMS and the tutorial. 

MEIMS captures maintenance error data, facilitates the identification of common 
maintenance errors and associated trends, and supports understanding of how to identify 
human errors in the future. The target audience for this information management system 
tool includes safety personnel (data entry & retrieval by unit safety officers, other safety 
& training personnel, maintenance officers, maintenance supervisors), mishap 
investigators-for data retrieval (Aircraft Mishap Board members, squadron safety 
officers), and analysts (from the Naval Safety Center, the command’s safety officer or 
one from its higher headquarters). This tool allows can directly lead to a decreased 
mishap rate and overall increased mission readiness due to the training and analysis 
opportunity it provides. 

Usability Study. You will be given a packet of instructions to guide you through 
MEIMS. You will be asked to make comments on the effectiveness and usability of the 
prototype system during your testing phase. Additionally, you will be asked to complete 
an “exit survey” after completion of your testing. Questions will include demographic 
information, objective questions about MEIMS usability, questions regarding the tutorial 
and its effectiveness (if applicable), and subjective questions and comments for areas not 
covered in the objective section. The study should take no more than 15-20 minutes. 

Completion of Study. Upon completion of your testing and survey you will be 
asked to return your packet of instructions to LCDR McCracken. Thank you again for 
your participation. 



Mike McCracken 
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Instructions for Maintenance Error Information Management System (MEIMS) 

Tool Evaluation 



Start-up 

1. Go to a computer lab with OFFICE 2000 loaded. Log on into the NPGS domain. 
Open Microsoft ACCESS 2000, then using the zip disk provided, open the file Hfac2000. 

Question 1: Did you have any problems accessing the program ? Y /N (circle one) 

If so, please describe: 



Main Menu 

2. You will now have the Main Menu displayed with the world famous Supersonic 
Hornet photo in the background. 

3. Note the six categories next to the command buttons on the bottom right portion of the 
screen. The system has “focus” on “Query Menu”. Note the information on this button in 
the bottom left gray buffer above the Windows Start button. Place the mouse pointer 
over the Query Menu box (don’t click, if you do, select <Back> on subsequent page) and 
note information that appears in the Text Box (both of these sources of information will 
be available throughout MEIMS). 

4. Select <Tab> and view the same information for the remaining four command buttons 
(note, if you select <Exit> you will have to re-enter the system (see step 1 above). 

Question 2: Is the terminology clear enough to understand what each of the four 
command buttons does? If not, what could be changed to make it clearer? 



5. Select (click or tab to & enter) <Query Menu> 



Query Menu 

6. Note there are two sections on the Query Menu. The left half of the screen has seven 
categories to help you define how you would like to view the mishap data. The right half 
of the screen has four command buttons. 

7. Select <Aircraft Model> 

8. Another form appears: “Query by Aircraft Type”. Select your type aircraft, then 
select <View Selections “Summary of Mishap” Form appears. Note, your aircraft 
selection has a blue background. Review the “Brief Description” of the mishap and the 
“Contributing Factors.” View 
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Question 3: What aircraft did you select? 

How many separate mishaps of that aircraft type are in the database ?_ 
View one of the mishaps. 

What are the level 3 codes & what do they mean? 

How did you find that info ? 

When you are through viewing the data, select <Close Form> 

Select another aircraft type, and view some of the mishaps (optional). 
When complete select <Back> on Query by Aircraft Model Form 

9. Select another category (your option) & view the data. 

Question 4: Which (if any) of the seven categories do you find useful? 



Which (if any) of the seven categories do you not find useful? 



10. Select <Multiple Criteria>. Create your own query using two or more criteria. 
Question 5: What Criteria did you choose? 



How many mishaps for your selection were in the database? 

* 

Did you find this function useful? Why or why not? 



11. Return to Query Menu. Select <HFACS-ME Summaryx 

Question 6: How many total mishaps are in the database? 

How many mishaps have a level one category of Worker Conditions ?_ 
How many mishaps have a level two category of Medical? 
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Further define the system by your aircraft model (or select another type). 

Question 7: What aircraft did you select? 

How many separate mishaps of that type aircraft are in the database? 

Conduct further queries as desired. When complete, return to Query Menu & return to 
Main Menu. 

1 1 . Select <Report Menu>. Review the six command buttons and their functions. 

12. Select <Mishap Distribution - Locations Review the report data.Find your type 
aircraft (or review another) in the report. <Close> the report & return to the Report 
Menu. 

Question 8: How many Detached mishaps are in the database? 

13. Select <A11 Mishaps -Chronological Listing>. Review data. <Close> the report when 
complete & return to Report Menu. Return to Main Menu. 

14. Select <Expert Graph Menu>. Follow directions. Create one graph with aircraft 
model (yours and 1 or 2 others) on the X-Axis and HFACS-ME Level One (all four 
codes) on the Y-Axis. 

Question 9: What aircraft did you select? 

Did you notice a difference in the level one codes between the aircraft (if so, what)? 



Return to <Expert Graph Menu>. Try more graphs as desired. When complete, return to 
Main Menu. 

15. Select <Add New Data>. Enter two mishaps to the database: 

Question 10: What are the Mishap Numbers for the data you are entering? 



Check to see if your entries were added to the database by Looking at the end of the 
Chronological Listing on the Report Menu (look for your Mishap Numbers). 

Question 11: Did you see your data in the Chronological Listing? 
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16. Return to Main Menu & Select Investigation tool. This selection is not yet fully 
functional. It will be used to “drill down” a specific mishap or types of mishap to 
determine if like factors occur. It will provide enhanced analysis and training benefit. 
You may choose multiple criteria, just as in query, to limit narrow your focus (optional). 

17. Return to Main Menu & Exit the Program. 

18. Please fill out the Exit Survey Questionnaire. 



119 



THIS PAGE INTENTIONALLY LEFT BLANK 



120 



APPENDIX D 



MAINTENANCE ERROR INFORMATION MANAGEMENT SYSTEM (MEIMS) 

EXIT SURVEY 

User’s Impression of the Maintenance Error Information Management System (MEIMS) 

Prototype Tool 

Purpose: This survey evaluates a user’s overall satisfaction of the Maintenance Error 
Information Management System (MEIMS) prototype tool. It consists of four parts. 

Part I: Demographic Information. Part I provides the user’s aviation 
background, computer experience, and availability of software and hardware systems 
used in the Navy and Marine Corps. 

Part II: User Satisfaction with the MEIMS Prototype Tool. Part II deals directly 
with user feedback as they use the prototype tool. 

Part III: User Overall Satisfaction with the MEIMS Prototype Tool. Part III 
allows users to give general feedback about the prototype tool. 

Part IV: (if applicable) User Satisfaction with the Tutorial. Part IV allows user 
to provide feedback about the Tutorial 



Part I. Demographic Information 

Follow the instructions after each numbered question or statement. 

1. I am attached to a command that primarily performs maintenance (military and/or 
civilian) at the: 

(Select one from the list and check the box) 

□ Squadron Level 

□ Intermediate Level (AIMD) 

□ Depot Level (NADEP) 

□ Command does not perform aircraft maintenance 

□ Other (describe if other) 

2. How long have you been using a computer? 

(Select one from the list and check the box) 

□ Less than one month 

□ One month to less than one year 

□ One year to less than two years 

□ Two years or more 
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3. What software do you normally use? 

(Check all boxes that apply) 

□ Microsoft Office (Word, PowerPoint, Excel, Access) 
What version? 

(Check all boxes that apply) 

□ 97 

□ 2000 

□ not sure of version 

□ other (describe if other) 

□ Lotus Smart Suite (Word Pro, Lotus 123...) 

What version? 

(Check all boxes that apply) 

□ 97 

□ 9.5 

□ not sure of version 

□ other (describe if other) 

□ Corel Word Perfect Office (Word Perfect, Quattro Pro...) 
What version? 

(Check all boxes that apply) 

□ Corel Office 7 

□ 2000 

□ not sure of version 

□ other (describe if other) 

□ Other (describe if other) 

4. What software application categories are you familiar with? 

(Check all boxes that apply) 

□ Word Processing (MS Word, Word Perfect, Word Pro...) 

□ Spreadsheet (Excel, Lotus 123, Quattro Pro...) 

□ Presentations (PowerPoint, Harvard Graphics...) 

□ Graphic Software (Corel Draw, Adobe Photoshop...) 

□ E-Mail (Outlook, Eudora, AOL...) 

□ Database (Access, DBase...) 
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5. What computer operating systems do you use? 
(Check all boxes that apply) 

□ Windows (3.1, 95, 98, 2000) 

□ Windows NT 

□ Macintosh 

□ UNIX 

□ Linux 

□ Other (describe if other) 



Part II. User Satisfaction with the Four Sections of the MEIMS Prototype Tool 

Select the category that best matches your impression of each of the below categories 
(and check the box). 



Strongly Agree 
Agree 



Neutral Disagree Strongly 

Disagree 



I feel the information 
on the MEIMS tool was 
in a logical form 



I found the MEIMS 
tool easy to navigate 



My tour of the MEIMS 
tool was very interesting 



The information presented on 
the MEIMS tool is relevant to 
maintenance operations 



The concept of the MEIMS 
tool is a good one. 
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Part III. User Overall Satisfaction with the MEIMS Prototype Tool 

Please make any comments on the MEIMS Prototype Tool not reflected in your 
comments in sections 1 and 2. 

The most positive aspects of the MEIMS prototype tool were: 



The most negative aspects of the MEIMS prototype tool were: 



I would make these changes (if any) to the MEIMS prototype tool: 



If you received the Tutorial, please continue on to Part IV. If you did not receive 
the Tutorial, you are complete. Thank you! Your participation is greatly 

appreciated! 
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Part IV. User Satisfaction with the Tutorial 

Select the category that best matches your impression of each of the below categories 
(and check the box). 

Strongly Agree Neutral Disagree Strongly 
Agree Disagree 

I feel the information 
in the Tutorial was 
beneficial 



The Tutorial made 
the evaluation easier 
to complete 



The Tutorial was interesting 



The information presented in 
the Tutorial is relevant to 
the MEIMS tool 



The concept of a tutorial 
is a good one. 



Please make any additional comments on the Tutorial. 
The most positive aspects of the Tutorial were: 
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The most negative aspects of the Tutorial were: 



I would make these changes (if any) to the Tutorial. 



Thank you! Your participation is greatly appreciated! 
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